2024-03-21T08:22:05.941 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T08:22:05.942 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} --archive /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126 --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/orig.config.yaml 2024-03-21T08:22:05.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T08:22:06.071 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126 branch: wip-vshankar-testing-20240320.053002 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} email: vshankar@redhat.com first_in_suite: false job_id: '7612126' kernel: flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20240320.053002 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240320.053002 sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8073 sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 sleep_before_teardown: 0 subset: 203/1024 suite: fs suite_branch: wip-vshankar-testing-20240320.053002 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqM1DoNJIt2li0uTk/Ne4B6R5ZL8dWRK89+IfGdoWYkOgN7cBCrLUdVrRidmATZeTEUnWuEUd9RvhI3Jf/GsMc= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVQ4EoPkwqcFmwRr8rgNQDfaP1UqmopUAgnGlaKXBO7f187Ba+V0DzAY2sUYrJcKh7VjUHBywJPbUrQgUqXU2w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-20_11:21:18 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T08:22:06.071 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa; will attempt to use it 2024-03-21T08:22:06.072 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks 2024-03-21T08:22:06.072 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T08:22:06.074 INFO:teuthology.task.internal:Checking packages... 2024-03-21T08:22:06.097 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '2edfee68ad7243bed422a049b6f5f58619300f11' 2024-03-21T08:22:06.098 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T08:22:06.098 INFO:teuthology.packaging:ref: None 2024-03-21T08:22:06.098 INFO:teuthology.packaging:tag: None 2024-03-21T08:22:06.098 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240320.053002 2024-03-21T08:22:06.098 INFO:teuthology.packaging:sha1: 2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T08:22:06.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20240320.053002 2024-03-21T08:22:06.249 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:22:06.251 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T08:22:06.258 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T08:22:06.258 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T08:22:06.299 INFO:teuthology.task.internal:Saving configuration 2024-03-21T08:22:06.315 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T08:22:06.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T08:22:06.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 08:16:03.359200', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvWc703ocFKliOEdmzfbTl03MMNUOu+QkGO7lCWHboK'} 2024-03-21T08:22:06.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 08:16:03.362281', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIxgl+/brFID+7RNW8VlCqNzhlC5a/eNS6dNAG2Jtsra'} 2024-03-21T08:22:06.370 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T08:22:06.376 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-21T08:22:06.376 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-21T08:22:06.376 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T08:22:06.473 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8963e964c0>, signals=[15]) 2024-03-21T08:22:06.473 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T08:22:06.480 INFO:teuthology.task.internal:Opening connections... 2024-03-21T08:22:06.480 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:22:06.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:22:06.562 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:22:06.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:22:06.641 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T08:22:06.774 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-03-21T08:22:06.780 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-03-21T08:22:06.781 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="Ubuntu" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="22.04" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_CODENAME=jammy 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:ID=ubuntu 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE=debian 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-21T08:22:06.828 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-21T08:22:06.829 INFO:teuthology.orchestra.run.smithi028.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-21T08:22:06.829 INFO:teuthology.orchestra.run.smithi028.stdout:UBUNTU_CODENAME=jammy 2024-03-21T08:22:06.829 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-03-21T08:22:06.852 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-21T08:22:06.858 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-21T08:22:06.858 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Ubuntu" 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="22.04" 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_CODENAME=jammy 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:ID=ubuntu 2024-03-21T08:22:06.907 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE=debian 2024-03-21T08:22:06.908 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-21T08:22:06.908 INFO:teuthology.orchestra.run.smithi151.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-21T08:22:06.908 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-21T08:22:06.908 INFO:teuthology.orchestra.run.smithi151.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-21T08:22:06.908 INFO:teuthology.orchestra.run.smithi151.stdout:UBUNTU_CODENAME=jammy 2024-03-21T08:22:06.908 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-21T08:22:06.935 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T08:22:06.945 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T08:22:06.966 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T08:22:06.966 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-03-21T08:22:06.969 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-21T08:22:06.973 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T08:22:06.992 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T08:22:06.992 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-03-21T08:22:07.017 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-21T08:22:07.033 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T08:22:07.414 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T08:22:07.429 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-21T08:22:07.429 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, timeout 300 2024-03-21T08:22:07.429 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T08:22:07.430 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T08:22:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:22:07.435 DEBUG:teuthology.task.kernel:sha1 for mon.a is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.435 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-21T08:22:07.435 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-21T08:22:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:22:07.436 DEBUG:teuthology.task.kernel:sha1 for mon.b is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.436 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-21T08:22:07.436 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-21T08:22:07.483 INFO:teuthology.orchestra.run.smithi151.stdout:5.15.0-56-generic 2024-03-21T08:22:07.483 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.483 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-21T08:22:07.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.487 INFO:teuthology.orchestra.run.smithi028.stdout:5.15.0-56-generic 2024-03-21T08:22:07.487 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.487 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-21T08:22:07.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:22:07.637 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-03-21T08:22:07.638 DEBUG:teuthology.orchestra.run.smithi151:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-21T08:22:07.663 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-21T08:22:07.682 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-03-21T08:22:07.683 DEBUG:teuthology.orchestra.run.smithi028:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-21T08:22:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-21T08:22:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-21T08:22:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-21T08:22:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-21T08:22:07.802 INFO:teuthology.orchestra.run.smithi151.stderr:hint: 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: git config --global init.defaultBranch 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: 2024-03-21T08:22:07.803 INFO:teuthology.orchestra.run.smithi151.stderr:hint: git branch -m 2024-03-21T08:22:07.806 INFO:teuthology.orchestra.run.smithi151.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-21T08:22:07.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-21T08:22:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: git config --global init.defaultBranch 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: 2024-03-21T08:22:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:hint: git branch -m 2024-03-21T08:22:07.836 INFO:teuthology.orchestra.run.smithi028.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-21T08:22:07.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-21T08:22:07.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-21T08:22:07.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-21T08:22:07.932 DEBUG:teuthology.orchestra.run.smithi151:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-21T08:22:07.956 DEBUG:teuthology.orchestra.run.smithi028:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-21T08:25:41.472 INFO:teuthology.orchestra.run.smithi151.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-21T08:25:41.472 INFO:teuthology.orchestra.run.smithi151.stderr: * [new branch] main -> origin/main 2024-03-21T08:25:41.504 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190312 -> 20190312 2024-03-21T08:25:41.504 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190416 -> 20190416 2024-03-21T08:25:41.505 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190514 -> 20190514 2024-03-21T08:25:41.505 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190618 -> 20190618 2024-03-21T08:25:41.505 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190717 -> 20190717 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190815 -> 20190815 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20190923 -> 20190923 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20191022 -> 20191022 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20191215 -> 20191215 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200122 -> 20200122 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200316 -> 20200316 2024-03-21T08:25:41.506 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200421 -> 20200421 2024-03-21T08:25:41.507 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200519 -> 20200519 2024-03-21T08:25:41.507 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200619 -> 20200619 2024-03-21T08:25:41.507 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200721 -> 20200721 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200817 -> 20200817 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20200918 -> 20200918 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20201022 -> 20201022 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20201118 -> 20201118 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20201218 -> 20201218 2024-03-21T08:25:41.508 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210208 -> 20210208 2024-03-21T08:25:41.509 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210315 -> 20210315 2024-03-21T08:25:41.509 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210511 -> 20210511 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210716 -> 20210716 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210818 -> 20210818 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20210919 -> 20210919 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20211027 -> 20211027 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20211216 -> 20211216 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220209 -> 20220209 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220310 -> 20220310 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220411 -> 20220411 2024-03-21T08:25:41.510 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220509 -> 20220509 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220610 -> 20220610 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220708 -> 20220708 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220815 -> 20220815 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20220913 -> 20220913 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20221012 -> 20221012 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20221109 -> 20221109 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20221214 -> 20221214 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230117 -> 20230117 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230210 -> 20230210 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230310 -> 20230310 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230404 -> 20230404 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230515 -> 20230515 2024-03-21T08:25:41.513 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230625 -> 20230625 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230804 -> 20230804 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20230919 -> 20230919 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20231030 -> 20231030 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20231111 -> 20231111 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20231211 -> 20231211 2024-03-21T08:25:41.514 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20240115 -> 20240115 2024-03-21T08:25:41.515 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20240220 -> 20240220 2024-03-21T08:25:41.515 INFO:teuthology.orchestra.run.smithi151.stderr: * [new tag] 20240312 -> 20240312 2024-03-21T08:25:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-21T08:25:42.681 INFO:teuthology.orchestra.run.smithi028.stderr: * [new branch] main -> origin/main 2024-03-21T08:25:42.718 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190312 -> 20190312 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190416 -> 20190416 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190514 -> 20190514 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190618 -> 20190618 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190717 -> 20190717 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190815 -> 20190815 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20190923 -> 20190923 2024-03-21T08:25:42.719 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20191022 -> 20191022 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20191215 -> 20191215 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200122 -> 20200122 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200316 -> 20200316 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200421 -> 20200421 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200519 -> 20200519 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200619 -> 20200619 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200721 -> 20200721 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200817 -> 20200817 2024-03-21T08:25:42.720 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20200918 -> 20200918 2024-03-21T08:25:42.721 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20201022 -> 20201022 2024-03-21T08:25:42.721 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20201118 -> 20201118 2024-03-21T08:25:42.721 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20201218 -> 20201218 2024-03-21T08:25:42.721 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210208 -> 20210208 2024-03-21T08:25:42.721 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210315 -> 20210315 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210511 -> 20210511 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210716 -> 20210716 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210818 -> 20210818 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20210919 -> 20210919 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20211027 -> 20211027 2024-03-21T08:25:42.722 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20211216 -> 20211216 2024-03-21T08:25:42.727 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220209 -> 20220209 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220310 -> 20220310 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220411 -> 20220411 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220509 -> 20220509 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220610 -> 20220610 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220708 -> 20220708 2024-03-21T08:25:42.728 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220815 -> 20220815 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20220913 -> 20220913 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20221012 -> 20221012 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20221109 -> 20221109 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20221214 -> 20221214 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230117 -> 20230117 2024-03-21T08:25:42.729 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230210 -> 20230210 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230310 -> 20230310 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230404 -> 20230404 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230515 -> 20230515 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230625 -> 20230625 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230804 -> 20230804 2024-03-21T08:25:42.730 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20230919 -> 20230919 2024-03-21T08:25:42.731 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20231030 -> 20231030 2024-03-21T08:25:42.731 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20231111 -> 20231111 2024-03-21T08:25:42.731 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20231211 -> 20231211 2024-03-21T08:25:42.731 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20240115 -> 20240115 2024-03-21T08:25:42.731 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20240220 -> 20240220 2024-03-21T08:25:42.732 INFO:teuthology.orchestra.run.smithi028.stderr: * [new tag] 20240312 -> 20240312 2024-03-21T08:25:45.672 INFO:teuthology.orchestra.run.smithi151.stderr:Updating files: 51% (1863/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 65% (2339/3598) Updating files: 65% (2351/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 80% (2887/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 95% (3422/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-21T08:25:45.677 INFO:teuthology.orchestra.run.smithi151.stdout:HEAD is now at 22ee1d8c Merge branch 'robot/pr-0-1710861164' into 'main' 2024-03-21T08:25:45.698 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-21T08:25:45.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:25:45.842 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:25:45.842 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ --input-file=- 2024-03-21T08:25:45.845 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi151.front.sepia.ceph.com to complete... 2024-03-21T08:25:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:Updating files: 50% (1811/3598) Updating files: 51% (1835/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 63% (2280/3598) Updating files: 64% (2303/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 71% (2580/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 83% (3011/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-21T08:25:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:HEAD is now at 22ee1d8c Merge branch 'robot/pr-0-1710861164' into 'main' 2024-03-21T08:25:47.124 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-21T08:25:47.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-21T08:25:47.280 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:25:47.281 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ --input-file=- 2024-03-21T08:25:47.283 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-03-21T08:25:47.355 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21 08:25:47 URL:https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb [39270258/39270258] -> "/tmp/linux-image.deb" [1] 2024-03-21T08:25:47.356 INFO:teuthology.orchestra.run.smithi151.stderr:FINISHED --2024-03-21 08:25:47-- 2024-03-21T08:25:47.356 INFO:teuthology.orchestra.run.smithi151.stderr:Total wall clock time: 1.5s 2024-03-21T08:25:47.356 INFO:teuthology.orchestra.run.smithi151.stderr:Downloaded: 1 files, 37M in 1.4s (27.5 MB/s) 2024-03-21T08:25:47.360 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-21T08:25:47.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo dpkg -i /tmp/linux-image.deb 2024-03-21T08:25:47.598 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-image-6.8.0-g5ec3e6e88b2e. 2024-03-21T08:25:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21 08:25:48 URL:https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb [39270258/39270258] -> "/tmp/linux-image.deb" [1] 2024-03-21T08:25:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-03-21 08:25:48-- 2024-03-21T08:25:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 1.5s 2024-03-21T08:25:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 37M in 1.4s (27.5 MB/s) 2024-03-21T08:25:48.807 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-21T08:25:48.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo dpkg -i /tmp/linux-image.deb 2024-03-21T08:25:49.062 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-image-6.8.0-g5ec3e6e88b2e. 2024-03-21T08:25:49.656 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-21T08:25:49.660 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-21T08:25:49.793 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-21T08:25:51.067 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-21T08:25:51.071 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-21T08:25:51.201 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-21T08:25:53.407 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-21T08:25:53.510 INFO:teuthology.orchestra.run.smithi151.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:25:54.813 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-21T08:25:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:14.857 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub' 2024-03-21T08:26:14.873 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-21T08:26:14.883 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-21T08:26:15.181 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:15.207 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:15.490 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-21T08:26:15.505 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-21T08:26:15.566 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-21T08:26:15.572 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-21T08:26:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-21T08:26:15.766 INFO:teuthology.orchestra.run.smithi151.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-21T08:26:15.767 INFO:teuthology.orchestra.run.smithi151.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-21T08:26:15.769 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub' 2024-03-21T08:26:15.785 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-21T08:26:15.812 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-21T08:26:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-03-21T08:26:15.997 DEBUG:teuthology.orchestra.run.smithi151:> dpkg-deb -c /tmp/linux-image.deb 2024-03-21T08:26:16.005 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./ 2024-03-21T08:26:16.006 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./boot/ 2024-03-21T08:26:16.006 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 4474370 2024-03-19 20:13 ./boot/System.map-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.018 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 170904 2024-03-19 20:13 ./boot/config-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.019 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11555840 2024-03-19 20:13 ./boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/ 2024-03-21T08:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postinst.d/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postrm.d/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/preinst.d/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/prerm.d/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ 2024-03-21T08:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 490449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-21T08:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-21T08:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-21T08:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 93081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 106697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 152145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 83857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-21T08:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/ 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/ 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/ 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpuid.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/msr.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/ 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 354265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-21T08:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 886177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-21T08:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 2294937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-21T08:26:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ 2024-03-21T08:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/af_alg.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_hash.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_skcipher.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/ 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/ 2024-03-21T08:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-21T08:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-21T08:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-21T08:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-21T08:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-21T08:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-21T08:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cryptd.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_simd.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-21T08:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lrw.ko 2024-03-21T08:26:16.141 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 143033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xts.ko 2024-03-21T08:26:16.143 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_extlog.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/ 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/einj.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-21T08:26:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 104433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 89937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 88313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-21T08:26:16.145 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-21T08:26:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_legacy.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-21T08:26:16.147 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-21T08:26:16.148 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_radisys.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rz1000.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-21T08:26:16.149 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 83257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-21T08:26:16.150 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/ 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 124457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/ 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/ 2024-03-21T08:26:16.151 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 113833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/ 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 964817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 169705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/ 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 94217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 116337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/ 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 110329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 98193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 263641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-21T08:26:16.152 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/ 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 98121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/ 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/applicom.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/ 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-21T08:26:16.153 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 148521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 139361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/ 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 84025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-21T08:26:16.154 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/nvram.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/ 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-21T08:26:16.155 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/ 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/ 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-palmas.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-si5351.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-twl6040.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-wm831x.ko 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/ 2024-03-21T08:26:16.156 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 246841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 570497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-21T08:26:16.157 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 122817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-21T08:26:16.158 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/ 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-gpio.ko 2024-03-21T08:26:16.159 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max14577.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max77693.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max8997.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-palmas.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/ 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/ 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/ 2024-03-21T08:26:16.160 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-arizona.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9052.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9055.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-dln2.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 77753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-generic.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-kempld.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max7300.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max730x.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max732x.ko 2024-03-21T08:26:16.161 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-vx855.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-21T08:26:16.162 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/ 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-a4tech.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-apple.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-belkin.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cherry.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-chicony.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 60665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cp2112.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cypress.ko 2024-03-21T08:26:16.163 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ezkey.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-generic.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-21T08:26:16.165 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kensington.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-21T08:26:16.166 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lg-g15.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 57977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-magicmouse.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-microsoft.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-monterey.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ntrig.ko 2024-03-21T08:26:16.167 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-21T08:26:16.168 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-21T08:26:16.169 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-21T08:26:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 143393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 269233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/ 2024-03-21T08:26:16.171 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 125641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 218881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/ 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 64305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 90113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 304953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-21T08:26:16.172 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 97889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-21T08:26:16.174 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 99513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-21T08:26:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc2103.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 94601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-21T08:26:16.176 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hih6130.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 140137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-21T08:26:16.177 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/jc42.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-21T08:26:16.178 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-21T08:26:16.179 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-21T08:26:16.180 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 119849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-21T08:26:16.181 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 82721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-21T08:26:16.182 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 82481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-21T08:26:16.183 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-21T08:26:16.184 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 43449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 79841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-21T08:26:16.185 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 71257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 63137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 71233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 119417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-21T08:26:16.186 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/ 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/ 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/ 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-21T08:26:16.187 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-21T08:26:16.188 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-21T08:26:16.189 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/ 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 273529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 883945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-21T08:26:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 324337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 97841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 282345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 493185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/ 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 420593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/ 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 741737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ 2024-03-21T08:26:16.191 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 296881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 153729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 169145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 266577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 111337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-21T08:26:16.192 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 128665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 129801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/evbug.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ff-memless.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/ 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/ 2024-03-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8323.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8333.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-21T08:26:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt1070.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt2160.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/stowaway.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/matrix-keymap.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/ 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-21T08:26:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 123201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 89393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 335921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/psmouse.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/ 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 109601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-21T08:26:16.196 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ct82c710.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/parkbd.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/pcips2.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ps2mult.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serport.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/ 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-88pm860x.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-adp5520.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-bd2802.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da903x.ko 2024-03-21T08:26:16.197 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da9052.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-gpio.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3533.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm355x.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3642.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp8788.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-max8997.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-mc13783.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca9532.ko 2024-03-21T08:26:16.198 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca955x.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca963x.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-tca6507.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm8350.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-21T08:26:16.199 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/ 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/ 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 647121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 93321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bufio.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 155825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 107937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-era.ko 2024-03-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-mirror.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 77881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-region-hash.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 109457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-snapshot.ko 2024-03-21T08:26:16.201 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 181777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-zero.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/ 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 199881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 100841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 126369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 368497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-21T08:26:16.202 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/ 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/ 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 178049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 63241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 123233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/ 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm800.ko 2024-03-21T08:26:16.203 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm805.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm80x.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona-i2c.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 123649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/bcm590xx.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/dln2.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/janz-cmodio.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/kempld-core.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-core.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-21T08:26:16.204 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lp3943.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/max8907.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/menf21bmc.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-21T08:26:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/retu-mfd.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/si476x-core.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps65010.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps6507x.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-21T08:26:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wm8994.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/ 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/ 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-21T08:26:16.207 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/core.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/ 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ds1682.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/dummy-irq.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/ 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/echo.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/ 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-21T08:26:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/ 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 152073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hmc6352.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ics932s401.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-21T08:26:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/ 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/ 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 126153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 313601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/phantom.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/ 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-21T08:26:16.210 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/ 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 183569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/ 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 491609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/ 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/ 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 105937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-21T08:26:16.211 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 459001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/ 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 130665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 68785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-21T08:26:16.212 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 64617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/ 2024-03-21T08:26:16.213 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 87457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 97201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 102497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 90209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 81321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/ 2024-03-21T08:26:16.214 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 176697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1434505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 128369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 131281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 314329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/ 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 323737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/ 2024-03-21T08:26:16.215 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 193593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 347217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1045449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 166145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-21T08:26:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/ 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 204249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/ 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 141201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/ 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 60673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/ 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 281217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-21T08:26:16.218 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/ 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 90305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 295697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 584353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 246041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-21T08:26:16.219 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 944545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 288529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 513673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 93489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 735169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 151745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 86993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-21T08:26:16.220 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/ 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 102441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 122609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/ 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 727481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-21T08:26:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 278881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-21T08:26:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1122297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 95577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/ 2024-03-21T08:26:16.348 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/ 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 151097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/ 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 136801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/ 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-21T08:26:16.349 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/ 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 227865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 87017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 672057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/ 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/ 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 83809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-21T08:26:16.350 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 194073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/ 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 104049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/ 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1067001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/ 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/ 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 63201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-21T08:26:16.351 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/ 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 68049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/ 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 495777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-21T08:26:16.352 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/ 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 103185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 166049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/ 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/ 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 92345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-21T08:26:16.353 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/ 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 79937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 83721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/ 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/ 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/ 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/defxx.ko 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/ 2024-03-21T08:26:16.354 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 217113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/ 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 237265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/ 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ 2024-03-21T08:26:16.355 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 85857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/ 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-21T08:26:16.356 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/ 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-21T08:26:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 81961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 76569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 99009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-21T08:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/ 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 43769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/plip.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-21T08:26:16.359 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sb1000.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/ 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 64561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/ 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 107233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-21T08:26:16.360 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 86825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/ 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 144361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/ 2024-03-21T08:26:16.361 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 307193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/ 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 144017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/ 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/ 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 267745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 103361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/ 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 131329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-21T08:26:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 85217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/ 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/ 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pci-stub.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-21T08:26:16.363 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/ 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 146153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 112793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/ 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/ 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-21T08:26:16.364 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/ 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/ 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/ 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/ 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/ 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 131945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-21T08:26:16.366 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/ 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-21T08:26:16.367 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-21T08:26:16.369 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/ 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9030_battery.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9052-battery.ko 2024-03-21T08:26:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/gpio-charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max14577_charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max17042_battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8903_charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8925_power.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/sbs-battery.ko 2024-03-21T08:26:16.371 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/test_power.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_power.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm8350_power.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/ 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/ 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/ 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-21T08:26:16.372 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/ 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9052.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9055.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9063.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 68481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-21T08:26:16.373 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-21T08:26:16.374 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8907.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8925.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8997.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8998.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-palmas.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-21T08:26:16.375 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-21T08:26:16.376 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 77529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 57721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 135345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/ 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 271441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 100473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/ 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 356865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 295929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/ 2024-03-21T08:26:16.377 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 189145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/ 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/ 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 246937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/ 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1171953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/ 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 238401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/ 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 115329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-21T08:26:16.378 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 308961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 76041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 146753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 75977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 212385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 60505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/ 2024-03-21T08:26:16.379 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 157729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fdomain.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/ 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 294953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 201729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/imm.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 43977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 265945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/ 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 317193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-21T08:26:16.380 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/ 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 315321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 145497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/ 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 216401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/ 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 2194089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/ 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 310689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-21T08:26:16.381 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/ 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 659633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/ 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 119489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/ 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 81561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/ 2024-03-21T08:26:16.382 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 366337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 124601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ppa.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/ 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1876753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 85193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/ 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 558953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qlogicfas408.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 261081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 173929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 280113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-21T08:26:16.383 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 97625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 115649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 164889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/xen-scsifront.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 129393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-21T08:26:16.384 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/ 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/ 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 736761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/ 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 931521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 122489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/ 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 75521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/ 2024-03-21T08:26:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/ 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-21T08:26:16.386 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 57777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/moxa.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/mxser.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-21T08:26:16.387 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/ 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/ 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_uart.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/ 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 101993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/rp2.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-21T08:26:16.388 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/ 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 545401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/ 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_mf624.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_netx.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-21T08:26:16.389 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/ 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/ 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 145793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/ 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 87009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-21T08:26:16.390 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 89769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ulpi.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/ 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/ 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 452825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/ 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/ 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 60329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-21T08:26:16.391 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 92409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 50945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 122257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 57633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-21T08:26:16.392 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/ 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-21T08:26:16.393 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 161233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/ 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-21T08:26:16.394 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 157617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 69617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 75969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 73017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 65881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-21T08:26:16.395 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 81409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 156337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 87193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811_cs.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-21T08:26:16.396 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/ 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cytherm.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ehset.ko 2024-03-21T08:26:16.397 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/lvstest.ko 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/ 2024-03-21T08:26:16.398 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 64745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbtest.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/ 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/usbmon.ko 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/ 2024-03-21T08:26:16.399 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 301633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/ 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-21T08:26:16.400 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/f81232.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 175105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 136985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 124745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-21T08:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 107433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/metro-usb.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-21T08:26:16.402 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 78129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mxuport.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 191825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-21T08:26:16.403 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-21T08:26:16.404 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 125529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usbserial.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 52761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-21T08:26:16.405 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 180329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-21T08:26:16.406 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/ 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 71025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 104065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/ 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/ 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 162865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 103673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 76441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/ 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 113665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-21T08:26:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 89057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/ 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/ 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-21T08:26:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/bd6107.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-21T08:26:16.409 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/ 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-21T08:26:16.410 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/ 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-21T08:26:16.411 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 103049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/acquirewdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/advantechwdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/dw_wdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-21T08:26:16.412 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-21T08:26:16.413 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/retu_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-21T08:26:16.414 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-21T08:26:16.415 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/ 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/ 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 176385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 71153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-scsiback.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/ 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/ 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 158281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-21T08:26:16.416 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/adfs.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 163417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1019225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/kafs.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 103225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/autofs4.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 64625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/ 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 41017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/bfs.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-21T08:26:16.417 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 3487553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 305897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1335721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 97769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/configfs.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/ 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-21T08:26:16.418 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 644249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/efs.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1527665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/freevxfs.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/ 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1231361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-21T08:26:16.419 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 141713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 223281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 193545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/hpfs.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 420449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 281705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/ 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 88937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-21T08:26:16.420 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 490505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 81457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 117377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1058377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv2.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 120193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 2041825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/ 2024-03-21T08:26:16.421 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/ 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1511665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/ 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 582529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/ 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-21T08:26:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ascii.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-21T08:26:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 94041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 142577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 165649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 118025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-21T08:26:16.424 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-21T08:26:16.425 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 220545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ntfs.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 244673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 440681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 2590849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-21T08:26:16.426 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/omfs.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 360953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 56945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/qnx4.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 46625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/qnx6.ko 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/ 2024-03-21T08:26:16.427 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_tree.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v1.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v2.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 532289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 2712601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/ 2024-03-21T08:26:16.428 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 134809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/ 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 91329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/ 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 248409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 162785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/ 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 4036193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/ 2024-03-21T08:26:16.429 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/ 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 80489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/locktorture.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/ 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/test_udelay.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 60265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc7.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/ 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-21T08:26:16.430 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/interval_tree_test.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/ 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/ 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/memory-notifier-error-inject.ko 2024-03-21T08:26:16.431 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/notifier-error-inject.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/percpu_test.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/pm-notifier-error-inject.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/ 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 143337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/rbtree_test.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/ 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-kstrtox.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-string_helpers.ko 2024-03-21T08:26:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 4863225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_bpf.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_firmware.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_module.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_user_copy.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/ 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/xz_dec_test.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/ 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-21T08:26:16.433 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/ 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 6905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/ 2024-03-21T08:26:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/ 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 82897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/ 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 198833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-21T08:26:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/ 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 667545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/ 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 790673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-21T08:26:16.436 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-21T08:26:16.437 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/ 2024-03-21T08:26:16.438 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1046825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/ 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 125705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/ 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 194505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 58745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/ 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 390089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/ 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 102073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/hsr.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 284721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 47881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-21T08:26:16.440 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 51217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 61513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 29081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ 2024-03-21T08:26:16.441 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-21T08:26:16.442 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-21T08:26:16.443 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-21T08:26:16.444 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-21T08:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 79929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_gre.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 94169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 54273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/mip6.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-21T08:26:16.446 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-21T08:26:16.447 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-21T08:26:16.448 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 74881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/ 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 84401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/ 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 125209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 25817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-21T08:26:16.449 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 53233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/ 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/lapb.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/ 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 268521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc2.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/ 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 309417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/ 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-21T08:26:16.450 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 119705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 99993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-21T08:26:16.451 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 102553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 102345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 111673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 59193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 112145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 112409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 110337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 113289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-21T08:26:16.452 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 412513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-21T08:26:16.453 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 411769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 118161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 108313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-21T08:26:16.454 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 92233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 96521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 44137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-21T08:26:16.455 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 623473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 42513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 22937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 40809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-21T08:26:16.456 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_osf.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_synproxy.ko 2024-03-21T08:26:16.457 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tproxy.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 89233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/x_tables.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 38777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-21T08:26:16.458 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-21T08:26:16.459 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-21T08:26:16.460 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 43201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-21T08:26:16.461 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 16817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-21T08:26:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 8457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpudp.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/ 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 14441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/ 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 12337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/ 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 384201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-21T08:26:16.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/ 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/ 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 112553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/phonet.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/pn_pep.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/ 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 253961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 165105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 68097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/ 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 828297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/rxrpc.ko 2024-03-21T08:26:16.464 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/ 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 37089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 43353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 19185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_cgroup.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-21T08:26:16.465 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 39521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 62657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 13321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 33425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 34593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq_codel.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 35457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-21T08:26:16.466 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 49849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 66449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 30225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 36153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 27409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 24777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 48537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 32817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-21T08:26:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 26793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 31321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 28273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/ 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 932945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/ 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/ 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 337009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 55041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-21T08:26:16.468 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 1513905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/ 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 704761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/ 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 867249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/ 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/ 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 87745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-21T08:26:16.469 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 67953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 98113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 11081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/ 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 131273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/x25.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/ 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_algo.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 20553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 105217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_user.ko 2024-03-21T08:26:16.470 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/ 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/ 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 15441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 642383 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 625850 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 7677 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 23498 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 9916 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 70732 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 147214 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-21T08:26:16.471 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 233213 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 210 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 72614 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 879 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 257955 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 312782 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/ 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/ 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/ 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:26:16.472 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 171 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/changelog.Debian.gz 2024-03-21T08:26:16.473 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r-- root/root 691 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/copyright 2024-03-21T08:26:16.479 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.479 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.479 DEBUG:teuthology.orchestra.run.smithi151:> egrep '(submenu|menuentry.*6.8.0-g5ec3e6e88b2e).*{' /boot/grub/grub.cfg 2024-03-21T08:26:16.481 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-21T08:26:16.481 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-21T08:26:16.489 INFO:teuthology.orchestra.run.smithi151.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:16.489 INFO:teuthology.orchestra.run.smithi151.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:16.489 INFO:teuthology.orchestra.run.smithi151.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:16.490 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-21T08:26:16.490 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.490 DEBUG:teuthology.orchestra.run.smithi151:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-21T08:26:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-21T08:26:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-21T08:26:16.535 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi151.front.sepia.ceph.com to complete... 2024-03-21T08:26:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-21T08:26:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-21T08:26:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-21T08:26:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-03-21T08:26:16.855 DEBUG:teuthology.orchestra.run.smithi028:> dpkg-deb -c /tmp/linux-image.deb 2024-03-21T08:26:16.863 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./ 2024-03-21T08:26:16.863 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./boot/ 2024-03-21T08:26:16.863 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 4474370 2024-03-19 20:13 ./boot/System.map-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.874 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 170904 2024-03-19 20:13 ./boot/config-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.874 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11555840 2024-03-19 20:13 ./boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postinst.d/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postrm.d/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/preinst.d/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/prerm.d/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/ 2024-03-21T08:26:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:26:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/ 2024-03-21T08:26:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/ 2024-03-21T08:26:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/ 2024-03-21T08:26:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ 2024-03-21T08:26:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 490449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-21T08:26:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-21T08:26:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-21T08:26:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-21T08:26:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 93081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-21T08:26:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 106697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 152145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 83857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-21T08:26:16.896 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/ 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/ 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/ 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpuid.ko 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/msr.ko 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/ 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 354265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-21T08:26:16.897 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 886177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-21T08:26:16.898 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 2294937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/af_alg.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_hash.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_skcipher.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-21T08:26:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/ 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/ 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-21T08:26:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cryptd.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_simd.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-21T08:26:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lrw.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-21T08:26:16.915 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 143033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xts.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_extlog.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-21T08:26:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/ 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/einj.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 104433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 89937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 88313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-21T08:26:16.917 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-21T08:26:16.918 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_legacy.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-21T08:26:16.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_radisys.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rz1000.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-21T08:26:16.920 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 83257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ 2024-03-21T08:26:16.921 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/ 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 124457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/ 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/ 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 113833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/ 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 964817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 169705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-21T08:26:16.922 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/ 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 94217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 116337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/ 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 110329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 98193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 263641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/ 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 98121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/ 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/applicom.ko 2024-03-21T08:26:16.923 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/ 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 148521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 139361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-21T08:26:16.924 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/ 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 84025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/nvram.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/ 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-21T08:26:16.925 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/ 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/ 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-palmas.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-si5351.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-twl6040.ko 2024-03-21T08:26:16.926 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-wm831x.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/ 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 246841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/ 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/ 2024-03-21T08:26:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-21T08:26:16.928 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 570497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-21T08:26:16.929 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub' 2024-03-21T08:26:16.929 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-21T08:26:16.940 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/ 2024-03-21T08:26:16.991 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/ 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/ 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 122817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-21T08:26:16.992 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/ 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-gpio.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max14577.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max77693.ko 2024-03-21T08:26:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max8997.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-palmas.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/ 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/ 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/ 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-21T08:26:16.994 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-arizona.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9052.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9055.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-dln2.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 77753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-generic.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-kempld.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max7300.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max730x.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max732x.ko 2024-03-21T08:26:16.995 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-vx855.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-21T08:26:16.996 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/ 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-a4tech.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-apple.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-belkin.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cherry.ko 2024-03-21T08:26:16.997 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-chicony.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 60665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cp2112.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cypress.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ezkey.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-generic.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-21T08:26:16.998 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kensington.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-21T08:26:16.999 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lg-g15.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 57977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-magicmouse.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-microsoft.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-monterey.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ntrig.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-21T08:26:17.000 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-21T08:26:17.001 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-21T08:26:17.002 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 143393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-21T08:26:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 269233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/ 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 125641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 218881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/ 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 64305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 90113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-21T08:26:17.004 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 304953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 97889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-21T08:26:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-21T08:26:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 99513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-21T08:26:17.007 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc2103.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 94601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-21T08:26:17.008 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hih6130.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 140137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/jc42.ko 2024-03-21T08:26:17.009 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-21T08:26:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-21T08:26:17.011 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-21T08:26:17.012 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 119849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 82721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-21T08:26:17.013 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 82481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-21T08:26:17.014 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-21T08:26:17.015 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 43449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 79841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 71257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 63137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-21T08:26:17.016 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 71233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 119417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/ 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/ 2024-03-21T08:26:17.017 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/ 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-21T08:26:17.018 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-21T08:26:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-21T08:26:17.020 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-21T08:26:17.021 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/ 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-21T08:26:17.022 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 273529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 883945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 324337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 97841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 282345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-21T08:26:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 493185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/ 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 420593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/ 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 741737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-21T08:26:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 296881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 153729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 169145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 266577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 111337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ 2024-03-21T08:26:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 128665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 129801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/evbug.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ff-memless.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/ 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/ 2024-03-21T08:26:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8323.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8333.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-21T08:26:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt1070.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt2160.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/stowaway.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/matrix-keymap.ko 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/ 2024-03-21T08:26:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 123201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 89393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 335921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/psmouse.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/ 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 109601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ 2024-03-21T08:26:17.029 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ct82c710.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/parkbd.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/pcips2.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ps2mult.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serport.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/ 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-88pm860x.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-adp5520.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-bd2802.ko 2024-03-21T08:26:17.030 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da903x.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da9052.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-gpio.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3533.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm355x.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3642.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp8788.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-max8997.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-mc13783.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-21T08:26:17.031 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca9532.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca955x.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca963x.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-tca6507.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm8350.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-21T08:26:17.032 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/ 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/ 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 647121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 93321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bufio.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 155825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 107937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-era.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-21T08:26:17.033 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-mirror.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 77881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-region-hash.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 109457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-snapshot.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-21T08:26:17.034 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 181777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-zero.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/ 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 199881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 100841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 126369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 368497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/ 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/ 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 178049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 63241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-21T08:26:17.035 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 123233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/ 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm800.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm805.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm80x.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona-i2c.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 123649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/bcm590xx.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-21T08:26:17.036 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/dln2.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/janz-cmodio.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/kempld-core.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-core.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lp3943.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/max8907.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/menf21bmc.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-21T08:26:17.037 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/retu-mfd.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/si476x-core.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps65010.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps6507x.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wm8994.ko 2024-03-21T08:26:17.038 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/ 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/ 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/core.ko 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/ 2024-03-21T08:26:17.039 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ds1682.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/dummy-irq.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/ 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/echo.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/ 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/ 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 152073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hmc6352.ko 2024-03-21T08:26:17.040 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ics932s401.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/ 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/ 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 126153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 313601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-21T08:26:17.041 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/phantom.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/ 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/ 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 183569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ 2024-03-21T08:26:17.042 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/ 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 491609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/ 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/ 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 105937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 459001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-21T08:26:17.043 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/ 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 130665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 68785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 64617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-21T08:26:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/ 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/ 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/ 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/ 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/ 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-21T08:26:17.045 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 87457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/ 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 97201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 102497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 90209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-21T08:26:17.046 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 81321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/ 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 176697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1434505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 128369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 131281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 314329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/ 2024-03-21T08:26:17.047 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 323737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 193593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 347217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1045449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 166145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-21T08:26:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/ 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 204249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/ 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-21T08:26:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 141201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/ 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 60673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/ 2024-03-21T08:26:17.050 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 281217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 90305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 295697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 584353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-21T08:26:17.051 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 246041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 944545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 288529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/ 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 513673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-21T08:26:17.052 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 93489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 735169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 151745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 86993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/ 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 102441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 122609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-21T08:26:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/ 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 727481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 278881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1122297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 95577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-21T08:26:17.054 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/ 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/ 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 151097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-21T08:26:17.055 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 136801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 227865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 87017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 672057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/ 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-21T08:26:17.056 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 83809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 194073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/ 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 104049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/ 2024-03-21T08:26:17.057 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1067001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-21T08:26:17.127 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.133 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/ 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/ 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 63201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/ 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-21T08:26:17.174 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 68049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/ 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 495777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-21T08:26:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/ 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 103185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 166049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/ 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/ 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 92345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/ 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 79937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 83721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/ 2024-03-21T08:26:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/ 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/ 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/defxx.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/ 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 217113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/ 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 237265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-21T08:26:17.177 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/ 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 85857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/ 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-21T08:26:17.178 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/ 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-21T08:26:17.179 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 81961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 76569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 99009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/ 2024-03-21T08:26:17.181 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 43769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/plip.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-21T08:26:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sb1000.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/ 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 64561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/ 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 107233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-21T08:26:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 86825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/ 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 144361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/ 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 307193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/ 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 144017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-21T08:26:17.184 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/ 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/ 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 267745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 103361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/ 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 131329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-21T08:26:17.185 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 85217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/ 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/ 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pci-stub.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/ 2024-03-21T08:26:17.186 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 146153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 112793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/ 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/ 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/ 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-21T08:26:17.187 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/ 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/ 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/ 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/ 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-21T08:26:17.189 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 131945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-21T08:26:17.190 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/ 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-21T08:26:17.191 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/ 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-21T08:26:17.192 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9030_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9052-battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/gpio-charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max14577_charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max17042_battery.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8903_charger.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8925_power.ko 2024-03-21T08:26:17.193 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/sbs-battery.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/test_power.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_power.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm8350_power.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/ 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/ 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/ 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-21T08:26:17.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/ 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9052.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9055.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9063.ko 2024-03-21T08:26:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 68481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-21T08:26:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8907.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8925.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8997.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8998.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-palmas.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-21T08:26:17.197 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-21T08:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 77529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 57721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 135345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/ 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 271441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 100473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/ 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 356865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-21T08:26:17.199 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 295929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 189145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 246937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1171953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 238401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/ 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 115329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-21T08:26:17.200 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/ 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 308961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/ 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 76041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 146753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 75977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/ 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 212385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-21T08:26:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 60505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/ 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 157729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fdomain.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/ 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 294953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 201729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/imm.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 43977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 265945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/ 2024-03-21T08:26:17.202 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 317193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/ 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 315321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 145497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/ 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 216401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/ 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 2194089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/ 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 310689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-21T08:26:17.203 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/ 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 659633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/ 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 119489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/ 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 81561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/ 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 366337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 124601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-21T08:26:17.204 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ppa.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/ 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1876753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 85193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/ 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 558953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qlogicfas408.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 261081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 173929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 280113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 97625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-21T08:26:17.205 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 115649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 164889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/xen-scsifront.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 129393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/ 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/ 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 736761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/ 2024-03-21T08:26:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 931521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 122489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/ 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 75521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/ 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/ 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-21T08:26:17.207 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-21T08:26:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 57777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/moxa.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/mxser.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/ 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/ 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_uart.ko 2024-03-21T08:26:17.209 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/ 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 101993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/rp2.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/ 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 545401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/ 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-21T08:26:17.210 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_mf624.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_netx.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/ 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/ 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 145793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-21T08:26:17.211 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/ 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 87009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 89769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ulpi.ko 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/ 2024-03-21T08:26:17.212 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/ 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 452825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/ 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/ 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 60329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 92409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-21T08:26:17.213 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 50945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 122257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 57633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/ 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-21T08:26:17.214 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-21T08:26:17.215 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 161233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/ 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 157617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 69617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 75969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 73017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-21T08:26:17.216 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 65881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 81409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 156337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 87193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811_cs.ko 2024-03-21T08:26:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/ 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cytherm.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ehset.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-21T08:26:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/lvstest.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/ 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 64745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-21T08:26:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbtest.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/ 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/usbmon.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/ 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 301633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/ 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-21T08:26:17.220 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-21T08:26:17.221 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/f81232.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 175105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 136985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 124745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 107433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-21T08:26:17.222 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/metro-usb.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 78129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mxuport.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-21T08:26:17.223 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 191825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-21T08:26:17.224 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 125529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usbserial.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-21T08:26:17.225 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 52761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-21T08:26:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 180329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/ 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 71025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 104065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/ 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/ 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 162865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-21T08:26:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 103673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 76441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/ 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 113665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 89057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/ 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/ 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-21T08:26:17.228 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/bd6107.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-21T08:26:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/ 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/ 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-21T08:26:17.230 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 103049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/acquirewdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/advantechwdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-21T08:26:17.231 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/dw_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-21T08:26:17.232 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/retu_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-21T08:26:17.233 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/ 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-21T08:26:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/ 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 176385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 71153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-scsiback.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/ 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/ 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 158281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/ 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/adfs.ko 2024-03-21T08:26:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 163417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1019225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/kafs.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 103225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/autofs4.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 64625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 41017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/bfs.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 3487553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/ 2024-03-21T08:26:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 305897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1335721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 97769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/configfs.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 644249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/efs.ko 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/ 2024-03-21T08:26:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1527665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/freevxfs.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1231361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 141713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 223281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/ 2024-03-21T08:26:17.238 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 193545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/hpfs.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 420449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 281705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 88937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 490505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/ 2024-03-21T08:26:17.239 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 81457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/ 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/ 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 117377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1058377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv2.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 120193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 2041825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/ 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/ 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1511665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/ 2024-03-21T08:26:17.240 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 582529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/ 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-21T08:26:17.241 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ascii.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-21T08:26:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 94041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 142577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 165649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 118025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-21T08:26:17.243 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ 2024-03-21T08:26:17.244 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 220545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ntfs.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 244673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 440681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 2590849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/ 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/omfs.ko 2024-03-21T08:26:17.245 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/ 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 360953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 56945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/ 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/qnx4.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/ 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 46625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/qnx6.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/ 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_tree.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v1.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v2.ko 2024-03-21T08:26:17.246 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 532289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 2712601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 134809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/ 2024-03-21T08:26:17.247 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 91329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 248409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 162785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 4036193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 80489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/locktorture.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/test_udelay.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 60265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ 2024-03-21T08:26:17.248 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc7.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/ 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/interval_tree_test.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/ 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-21T08:26:17.249 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/ 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/memory-notifier-error-inject.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/notifier-error-inject.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/percpu_test.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/pm-notifier-error-inject.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/ 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 143337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/rbtree_test.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/ 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-kstrtox.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-string_helpers.ko 2024-03-21T08:26:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 4863225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_bpf.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_firmware.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_module.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_user_copy.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/ 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/xz_dec_test.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/ 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/ 2024-03-21T08:26:17.251 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 6905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/ 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-21T08:26:17.252 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/ 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 82897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/ 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 198833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/ 2024-03-21T08:26:17.253 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 667545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/ 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 790673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-21T08:26:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-21T08:26:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/ 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1046825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/ 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 125705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/ 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 194505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 58745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/ 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 390089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-21T08:26:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/ 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 102073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/hsr.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 284721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 47881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ 2024-03-21T08:26:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 51217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 61513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 29081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-21T08:26:17.258 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-21T08:26:17.259 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-21T08:26:17.260 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-21T08:26:17.261 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 79929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_gre.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 94169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-21T08:26:17.262 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 54273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/mip6.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-21T08:26:17.263 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-21T08:26:17.264 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 74881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/ 2024-03-21T08:26:17.265 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 84401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/ 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 125209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 25817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 53233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/ 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/lapb.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/ 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-21T08:26:17.266 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 268521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc2.ko 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/ 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 309417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/ 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 119705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-21T08:26:17.267 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 99993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 102553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 102345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 111673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 59193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 112145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 112409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 110337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-21T08:26:17.268 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 113289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 412513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-21T08:26:17.269 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 411769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-21T08:26:17.270 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 118161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 108313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-21T08:26:17.271 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 92233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 96521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-21T08:26:17.272 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 44137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 623473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 42513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 22937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-21T08:26:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 40809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_osf.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-21T08:26:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_synproxy.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tproxy.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 89233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/x_tables.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-21T08:26:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 38777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-21T08:26:17.276 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-21T08:26:17.277 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 43201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-21T08:26:17.278 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-21T08:26:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 16817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 8457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpudp.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-21T08:26:17.280 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/ 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 14441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/ 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 12337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/ 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 384201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/ 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/ 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 112553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/phonet.ko 2024-03-21T08:26:17.281 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/pn_pep.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/ 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 253961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 165105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 68097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/ 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 828297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/rxrpc.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/ 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 37089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 43353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-21T08:26:17.282 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 19185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_cgroup.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 39521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 62657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 13321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 33425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-21T08:26:17.283 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 34593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq_codel.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 35457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 49849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 66449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-21T08:26:17.284 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 30225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 36153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 27409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 24777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 48537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 32817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 26793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 31321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 28273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/ 2024-03-21T08:26:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 932945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/ 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/ 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 337009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 55041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 1513905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/ 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 704761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/ 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 867249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/ 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-21T08:26:17.286 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/ 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 87745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 67953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 98113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 11081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/ 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 131273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/x25.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/ 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_algo.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 20553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 105217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_user.ko 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/ 2024-03-21T08:26:17.287 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/ 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 15441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 642383 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 625850 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 7677 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 23498 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 9916 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 70732 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 147214 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 233213 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 210 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 72614 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 879 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 257955 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-21T08:26:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 312782 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/ 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/ 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/ 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/ 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 171 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/changelog.Debian.gz 2024-03-21T08:26:17.289 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r-- root/root 691 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/copyright 2024-03-21T08:26:17.293 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.293 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.293 DEBUG:teuthology.orchestra.run.smithi028:> egrep '(submenu|menuentry.*6.8.0-g5ec3e6e88b2e).*{' /boot/grub/grub.cfg 2024-03-21T08:26:17.300 INFO:teuthology.orchestra.run.smithi028.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:17.300 INFO:teuthology.orchestra.run.smithi028.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:17.300 INFO:teuthology.orchestra.run.smithi028.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-21T08:26:17.301 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-21T08:26:17.301 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.301 DEBUG:teuthology.orchestra.run.smithi028:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-21T08:26:17.343 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-03-21T08:26:17.383 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-21T08:26:17.389 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-21T08:26:17.432 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-21T08:26:17.437 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-21T08:26:17.564 INFO:teuthology.orchestra.run.smithi151.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-21T08:26:17.565 INFO:teuthology.orchestra.run.smithi151.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-21T08:26:17.565 INFO:teuthology.orchestra.run.smithi151.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-21T08:26:17.589 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-21T08:26:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub' 2024-03-21T08:26:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-21T08:26:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-03-21T08:26:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:17.996 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-21T08:26:18.229 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-21T08:26:18.233 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-21T08:26:18.274 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-21T08:26:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-21T08:26:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-21T08:26:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-21T08:26:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-21T08:26:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-03-21T08:26:48.763 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T08:26:48.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:26:48.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:26:49.492 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T08:26:49.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:26:49.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:27:07.935 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-03-21T08:27:16.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:27:16.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:27:23.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:27:32.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:27:32.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:27:35.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-03-21T08:27:35.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:27:47.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:27:47.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:27:47.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:27:47.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:27:49.375 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-03-21T08:27:50.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:28:04.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:28:04.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:28:04.598 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-21T08:28:05.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:28:05.313 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g5ec3e6e88b2e"... 2024-03-21T08:28:05.313 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-21T08:28:05.363 INFO:teuthology.orchestra.run.smithi028.stdout:6.8.0-g5ec3e6e88b2e 2024-03-21T08:28:05.363 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-21T08:28:05.363 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T08:28:05.363 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-21T08:28:05.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:28:05.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:28:06.364 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T08:28:06.364 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T08:28:06.456 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-03-21T08:28:06.507 DEBUG:teuthology.parallel:result is None 2024-03-21T08:28:08.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:28:26.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:28:26.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:28:29.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:28:50.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:28:50.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:28:53.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-21T08:29:17.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:29:17.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:29:17.946 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-21T08:29:18.688 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:29:18.688 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g5ec3e6e88b2e"... 2024-03-21T08:29:18.688 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-21T08:29:18.735 INFO:teuthology.orchestra.run.smithi151.stdout:6.8.0-g5ec3e6e88b2e 2024-03-21T08:29:18.735 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-21T08:29:18.736 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T08:29:18.736 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-21T08:29:19.736 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T08:29:19.736 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T08:29:19.814 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-21T08:29:19.866 DEBUG:teuthology.parallel:result is None 2024-03-21T08:29:19.866 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T08:29:19.875 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T08:29:19.875 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T08:29:19.879 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T08:29:19.888 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T08:29:19.915 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T08:29:19.948 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T08:29:19.949 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T08:29:19.952 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T08:29:20.008 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T08:29:20.017 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T08:29:20.018 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T08:29:20.021 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T08:29:20.054 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T08:29:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T08:29:20.067 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T08:29:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T08:29:20.077 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T08:29:20.087 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T08:29:20.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T08:29:20.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T08:29:20.147 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T08:29:20.159 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T08:29:20.159 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T08:29:20.193 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T08:29:20.201 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T08:29:20.250 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T08:29:20.303 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:29:20.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T08:29:20.378 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T08:29:20.388 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T08:29:20.437 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:29:20.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T08:29:20.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-03-21T08:29:20.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-21T08:29:20.650 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T08:29:20.661 INFO:teuthology.task.internal:Starting timer... 2024-03-21T08:29:20.661 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T08:29:20.718 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T08:29:20.748 DEBUG:teuthology.task.selinux:Excluding smithi028: OS 'ubuntu' does not support SELinux 2024-03-21T08:29:20.749 DEBUG:teuthology.task.selinux:Excluding smithi151: OS 'ubuntu' does not support SELinux 2024-03-21T08:29:20.749 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T08:29:20.749 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-21T08:29:20.749 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T08:29:20.749 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T08:29:20.784 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-21T08:29:20.790 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-21T08:29:20.912 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T08:29:20.929 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T08:29:20.930 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T08:37:14.773 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-03-21T08:37:14.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:37:14.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:37:14.865 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-21T08:37:15.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-21T08:37:15.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:37:15.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T08:37:15.090 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-21T08:37:15.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-21T08:37:15.205 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T08:37:15.217 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T08:37:15.217 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T08:37:15.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T08:37:15.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T08:37:15.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T08:37:15.259 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Command line: ntpd -gq 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: ---------------------------------------------------- 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: ntp-4 is maintained by Network Time Foundation, 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: corporation. Support and training for ntp-4 are 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: available at https://www.nwtime.org/support 2024-03-21T08:37:15.260 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: ---------------------------------------------------- 2024-03-21T08:37:15.261 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: proto: precision = 0.043 usec (-24) 2024-03-21T08:37:15.261 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: basedate set to 2022-02-04 2024-03-21T08:37:15.261 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: gps base set to 2022-02-06 (week 2196) 2024-03-21T08:37:15.261 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-21T08:37:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-21T08:37:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-21T08:37:15.262 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: restrict ::: KOD does nothing without LIMITED. 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen and drop on 0 v6wildcard [::]:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen normally on 2 lo 127.0.0.1:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen normally on 3 ens1f0 172.21.15.28:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen normally on 4 lo [::1]:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:ca3c%4]:123 2024-03-21T08:37:15.263 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:15 ntpd[17070]: Listening on routing socket on fd #22 for interface updates 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Command line: ntpd -gq 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: ---------------------------------------------------- 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: ntp-4 is maintained by Network Time Foundation, 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: corporation. Support and training for ntp-4 are 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: available at https://www.nwtime.org/support 2024-03-21T08:37:15.285 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: ---------------------------------------------------- 2024-03-21T08:37:15.286 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: proto: precision = 0.041 usec (-24) 2024-03-21T08:37:15.286 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: basedate set to 2022-02-04 2024-03-21T08:37:15.286 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: gps base set to 2022-02-06 (week 2196) 2024-03-21T08:37:15.286 INFO:teuthology.orchestra.run.smithi151.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-21T08:37:15.287 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-21T08:37:15.287 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: restrict ::: KOD does nothing without LIMITED. 2024-03-21T08:37:15.287 INFO:teuthology.orchestra.run.smithi151.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen and drop on 0 v6wildcard [::]:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen normally on 2 lo 127.0.0.1:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen normally on 3 enp3s0f1 172.21.15.151:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen normally on 4 lo [::1]:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f63%5]:123 2024-03-21T08:37:15.288 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:15 ntpd[17166]: Listening on routing socket on fd #22 for interface updates 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:16.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:16 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:17 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:18 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:19.260 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:19 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:19.260 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:19 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:19.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:19.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:19 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:20 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:20.285 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.285 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:20.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:20 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:21.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:21.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:21.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:21.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:21 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.262 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stdout:21 Mar 08:37:22 ntpd[17070]: ntpd: time slew -0.000601 s 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stdout:ntpd: time slew -0.000601s 2024-03-21T08:37:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-21T08:37:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:21 Mar 08:37:22 ntpd[17070]: can't open /var/log/ntpstats/loopstats.20240321: Permission denied 2024-03-21T08:37:22.285 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.285 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:22.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:22.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:22.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:22 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2024-03-21T08:37:22.361 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2024-03-21T08:37:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:22.362 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:23.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:23.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:23.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:23.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:23.286 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20240321: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20240321: Permission denied 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stdout:21 Mar 08:37:23 ntpd[17166]: ntpd: time slew -0.000165 s 2024-03-21T08:37:23.287 INFO:teuthology.orchestra.run.smithi151.stdout:ntpd: time slew -0.000165s 2024-03-21T08:37:23.288 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-21T08:37:23.288 INFO:teuthology.orchestra.run.smithi151.stderr:21 Mar 08:37:23 ntpd[17166]: can't open /var/log/ntpstats/loopstats.20240321: Permission denied 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:37:23.390 INFO:teuthology.run_tasks:Running task install... 2024-03-21T08:37:23.400 DEBUG:teuthology.task.install:project ceph 2024-03-21T08:37:23.400 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}} 2024-03-21T08:37:23.400 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'} 2024-03-21T08:37:23.400 INFO:teuthology.task.install:Using flavor: default 2024-03-21T08:37:23.406 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-21T08:37:23.406 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-21T08:37:23.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-key list | grep Ceph 2024-03-21T08:37:23.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-key list | grep Ceph 2024-03-21T08:37:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-21T08:37:23.540 INFO:teuthology.orchestra.run.smithi151.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-21T08:37:23.568 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-21T08:37:23.568 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph.com (release key) 2024-03-21T08:37:23.569 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-21T08:37:23.569 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-21T08:37:23.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T08:37:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-21T08:37:23.591 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph.com (release key) 2024-03-21T08:37:23.592 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-21T08:37:23.592 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-21T08:37:23.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T08:37:23.727 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default/ 2024-03-21T08:37:23.727 INFO:teuthology.task.install.deb:Package version is 19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:37:23.733 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default/ 2024-03-21T08:37:23.733 INFO:teuthology.task.install.deb:Package version is 19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:37:23.837 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:37:23.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-21T08:37:23.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2024-03-21T08:37:23.864 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:37:23.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-21T08:37:23.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2024-03-21T08:37:24.051 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-21T08:37:24.063 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-21T08:37:24.080 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-21T08:37:24.088 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-21T08:37:24.093 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy InRelease 2024-03-21T08:37:24.098 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-21T08:37:24.114 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-21T08:37:24.119 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-21T08:37:24.136 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-21T08:37:24.144 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy InRelease 2024-03-21T08:37:24.158 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-21T08:37:24.169 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-21T08:37:24.169 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-21T08:37:24.189 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-21T08:37:24.190 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2024-03-21T08:37:24.193 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-21T08:37:24.217 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,403 B] 2024-03-21T08:37:24.253 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-21T08:37:24.659 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 28.9 kB in 1s (50.2 kB/s) 2024-03-21T08:37:24.665 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 28.9 kB in 1s (51.8 kB/s) 2024-03-21T08:37:25.636 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-03-21T08:37:25.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2231-g2edfee68-1jammy cephadm=19.0.0-2231-g2edfee68-1jammy ceph-mds=19.0.0-2231-g2edfee68-1jammy ceph-mgr=19.0.0-2231-g2edfee68-1jammy ceph-common=19.0.0-2231-g2edfee68-1jammy ceph-fuse=19.0.0-2231-g2edfee68-1jammy ceph-test=19.0.0-2231-g2edfee68-1jammy ceph-volume=19.0.0-2231-g2edfee68-1jammy radosgw=19.0.0-2231-g2edfee68-1jammy python3-rados=19.0.0-2231-g2edfee68-1jammy python3-rgw=19.0.0-2231-g2edfee68-1jammy python3-cephfs=19.0.0-2231-g2edfee68-1jammy python3-rbd=19.0.0-2231-g2edfee68-1jammy libcephfs2=19.0.0-2231-g2edfee68-1jammy libcephfs-dev=19.0.0-2231-g2edfee68-1jammy librados2=19.0.0-2231-g2edfee68-1jammy librbd1=19.0.0-2231-g2edfee68-1jammy rbd-fuse=19.0.0-2231-g2edfee68-1jammy python3-cephfs=19.0.0-2231-g2edfee68-1jammy cephfs-shell=19.0.0-2231-g2edfee68-1jammy cephfs-top=19.0.0-2231-g2edfee68-1jammy cephfs-mirror=19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:37:25.694 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-03-21T08:37:25.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2231-g2edfee68-1jammy cephadm=19.0.0-2231-g2edfee68-1jammy ceph-mds=19.0.0-2231-g2edfee68-1jammy ceph-mgr=19.0.0-2231-g2edfee68-1jammy ceph-common=19.0.0-2231-g2edfee68-1jammy ceph-fuse=19.0.0-2231-g2edfee68-1jammy ceph-test=19.0.0-2231-g2edfee68-1jammy ceph-volume=19.0.0-2231-g2edfee68-1jammy radosgw=19.0.0-2231-g2edfee68-1jammy python3-rados=19.0.0-2231-g2edfee68-1jammy python3-rgw=19.0.0-2231-g2edfee68-1jammy python3-cephfs=19.0.0-2231-g2edfee68-1jammy python3-rbd=19.0.0-2231-g2edfee68-1jammy libcephfs2=19.0.0-2231-g2edfee68-1jammy libcephfs-dev=19.0.0-2231-g2edfee68-1jammy librados2=19.0.0-2231-g2edfee68-1jammy librbd1=19.0.0-2231-g2edfee68-1jammy rbd-fuse=19.0.0-2231-g2edfee68-1jammy python3-cephfs=19.0.0-2231-g2edfee68-1jammy cephfs-shell=19.0.0-2231-g2edfee68-1jammy cephfs-top=19.0.0-2231-g2edfee68-1jammy cephfs-mirror=19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:37:25.724 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-03-21T08:37:25.784 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-03-21T08:37:25.893 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2024-03-21T08:37:25.894 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2024-03-21T08:37:25.959 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2024-03-21T08:37:25.960 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2024-03-21T08:37:26.054 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages were automatically installed and are no longer required: 2024-03-21T08:37:26.055 INFO:teuthology.orchestra.run.smithi151.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-21T08:37:26.055 INFO:teuthology.orchestra.run.smithi151.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-21T08:37:26.055 INFO:teuthology.orchestra.run.smithi151.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-21T08:37:26.055 INFO:teuthology.orchestra.run.smithi151.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-21T08:37:26.056 INFO:teuthology.orchestra.run.smithi151.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-21T08:37:26.058 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-21T08:37:26.059 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout:Recommended packages: 2024-03-21T08:37:26.060 INFO:teuthology.orchestra.run.smithi151.stdout: btrfs-tools 2024-03-21T08:37:26.125 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2024-03-21T08:37:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-21T08:37:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-21T08:37:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-21T08:37:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-21T08:37:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-21T08:37:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-21T08:37:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-21T08:37:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-21T08:37:26.131 INFO:teuthology.orchestra.run.smithi028.stdout:Recommended packages: 2024-03-21T08:37:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: btrfs-tools 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-21T08:37:26.209 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-21T08:37:26.211 INFO:teuthology.orchestra.run.smithi151.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-21T08:37:26.212 INFO:teuthology.orchestra.run.smithi151.stdout: socat unzip xmlstarlet zip 2024-03-21T08:37:26.290 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2024-03-21T08:37:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-21T08:37:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-21T08:37:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-21T08:37:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-21T08:37:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-21T08:37:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-21T08:37:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-21T08:37:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-21T08:37:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-21T08:37:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-21T08:37:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: socat unzip xmlstarlet zip 2024-03-21T08:37:26.383 INFO:teuthology.orchestra.run.smithi151.stdout:0 upgraded, 124 newly installed, 0 to remove and 288 not upgraded. 2024-03-21T08:37:26.383 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 178 MB of archives. 2024-03-21T08:37:26.384 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 738 MB of additional disk space will be used. 2024-03-21T08:37:26.384 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2231-g2edfee68-1jammy [3,556 kB] 2024-03-21T08:37:26.406 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-21T08:37:26.435 INFO:teuthology.orchestra.run.smithi028.stdout:0 upgraded, 124 newly installed, 0 to remove and 288 not upgraded. 2024-03-21T08:37:26.435 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 178 MB of archives. 2024-03-21T08:37:26.435 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 738 MB of additional disk space will be used. 2024-03-21T08:37:26.435 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2231-g2edfee68-1jammy [3,556 kB] 2024-03-21T08:37:26.489 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-21T08:37:26.534 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-21T08:37:26.548 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-21T08:37:26.630 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-21T08:37:26.676 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-21T08:37:26.737 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-21T08:37:26.808 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2231-g2edfee68-1jammy [3,223 kB] 2024-03-21T08:37:26.831 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-21T08:37:26.845 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-21T08:37:26.847 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2231-g2edfee68-1jammy [3,223 kB] 2024-03-21T08:37:26.881 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-21T08:37:26.896 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-21T08:37:26.981 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-21T08:37:26.982 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2231-g2edfee68-1jammy [712 kB] 2024-03-21T08:37:26.993 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-21T08:37:27.023 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2231-g2edfee68-1jammy [356 kB] 2024-03-21T08:37:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2231-g2edfee68-1jammy [32.8 kB] 2024-03-21T08:37:27.043 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2231-g2edfee68-1jammy [180 kB] 2024-03-21T08:37:27.046 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-21T08:37:27.056 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2231-g2edfee68-1jammy [64.8 kB] 2024-03-21T08:37:27.059 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2231-g2edfee68-1jammy [330 kB] 2024-03-21T08:37:27.070 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-21T08:37:27.075 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2231-g2edfee68-1jammy [6,659 kB] 2024-03-21T08:37:27.080 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-21T08:37:27.087 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2231-g2edfee68-1jammy [712 kB] 2024-03-21T08:37:27.107 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-21T08:37:27.115 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-21T08:37:27.141 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2231-g2edfee68-1jammy [356 kB] 2024-03-21T08:37:27.169 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2231-g2edfee68-1jammy [32.8 kB] 2024-03-21T08:37:27.171 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2231-g2edfee68-1jammy [180 kB] 2024-03-21T08:37:27.188 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2231-g2edfee68-1jammy [64.8 kB] 2024-03-21T08:37:27.192 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2231-g2edfee68-1jammy [330 kB] 2024-03-21T08:37:27.216 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2231-g2edfee68-1jammy [6,659 kB] 2024-03-21T08:37:27.221 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-21T08:37:27.230 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-21T08:37:27.258 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-21T08:37:27.266 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-21T08:37:27.280 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-21T08:37:27.430 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-21T08:37:27.455 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2231-g2edfee68-1jammy [112 kB] 2024-03-21T08:37:27.460 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2231-g2edfee68-1jammy [467 kB] 2024-03-21T08:37:27.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2231-g2edfee68-1jammy [25.1 MB] 2024-03-21T08:37:27.741 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2231-g2edfee68-1jammy [112 kB] 2024-03-21T08:37:27.746 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2231-g2edfee68-1jammy [467 kB] 2024-03-21T08:37:27.787 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2231-g2edfee68-1jammy [25.1 MB] 2024-03-21T08:37:27.865 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-21T08:37:27.880 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-21T08:37:27.896 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-21T08:37:27.905 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-21T08:37:27.906 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-21T08:37:27.922 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-21T08:37:27.960 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-21T08:37:27.973 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-21T08:37:27.977 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-21T08:37:27.978 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-21T08:37:28.013 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-21T08:37:28.028 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-21T08:37:28.043 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-21T08:37:28.052 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-21T08:37:28.052 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-21T08:37:28.069 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-21T08:37:28.106 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-21T08:37:28.120 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-21T08:37:28.124 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-21T08:37:28.124 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-21T08:37:28.148 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-21T08:37:28.231 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-21T08:37:28.245 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-21T08:37:28.294 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-21T08:37:28.312 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-21T08:37:28.341 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-21T08:37:28.358 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-21T08:37:28.376 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-21T08:37:28.394 INFO:teuthology.orchestra.run.smithi151.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-21T08:37:28.396 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-21T08:37:28.457 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-21T08:37:28.496 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-21T08:37:28.517 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-21T08:37:28.542 INFO:teuthology.orchestra.run.smithi028.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-21T08:37:28.568 INFO:teuthology.orchestra.run.smithi151.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-21T08:37:28.582 INFO:teuthology.orchestra.run.smithi151.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-21T08:37:28.589 INFO:teuthology.orchestra.run.smithi151.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-21T08:37:28.594 INFO:teuthology.orchestra.run.smithi151.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-21T08:37:28.712 INFO:teuthology.orchestra.run.smithi028.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-21T08:37:28.727 INFO:teuthology.orchestra.run.smithi028.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-21T08:37:28.734 INFO:teuthology.orchestra.run.smithi028.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-21T08:37:28.740 INFO:teuthology.orchestra.run.smithi028.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-21T08:37:28.778 INFO:teuthology.orchestra.run.smithi151.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-21T08:37:28.860 INFO:teuthology.orchestra.run.smithi151.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-21T08:37:28.876 INFO:teuthology.orchestra.run.smithi151.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-21T08:37:28.912 INFO:teuthology.orchestra.run.smithi028.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2231-g2edfee68-1jammy [5,084 kB] 2024-03-21T08:37:28.923 INFO:teuthology.orchestra.run.smithi028.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-21T08:37:29.006 INFO:teuthology.orchestra.run.smithi028.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-21T08:37:29.021 INFO:teuthology.orchestra.run.smithi028.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-21T08:37:29.056 INFO:teuthology.orchestra.run.smithi151.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-21T08:37:29.107 INFO:teuthology.orchestra.run.smithi151.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-21T08:37:29.119 INFO:teuthology.orchestra.run.smithi151.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-21T08:37:29.190 INFO:teuthology.orchestra.run.smithi028.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2231-g2edfee68-1jammy [244 kB] 2024-03-21T08:37:29.204 INFO:teuthology.orchestra.run.smithi028.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2231-g2edfee68-1jammy [124 kB] 2024-03-21T08:37:29.206 INFO:teuthology.orchestra.run.smithi028.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-21T08:37:29.213 INFO:teuthology.orchestra.run.smithi028.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2231-g2edfee68-1jammy [1,525 kB] 2024-03-21T08:37:29.257 INFO:teuthology.orchestra.run.smithi028.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-21T08:37:29.270 INFO:teuthology.orchestra.run.smithi028.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-21T08:37:29.274 INFO:teuthology.orchestra.run.smithi151.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-21T08:37:29.275 INFO:teuthology.orchestra.run.smithi151.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-21T08:37:29.281 INFO:teuthology.orchestra.run.smithi151.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-21T08:37:29.285 INFO:teuthology.orchestra.run.smithi151.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-21T08:37:29.299 INFO:teuthology.orchestra.run.smithi151.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-21T08:37:29.303 INFO:teuthology.orchestra.run.smithi028.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2231-g2edfee68-1jammy [6,253 kB] 2024-03-21T08:37:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-21T08:37:29.428 INFO:teuthology.orchestra.run.smithi028.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-21T08:37:29.433 INFO:teuthology.orchestra.run.smithi028.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-21T08:37:29.437 INFO:teuthology.orchestra.run.smithi028.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-21T08:37:29.452 INFO:teuthology.orchestra.run.smithi028.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-21T08:37:29.483 INFO:teuthology.orchestra.run.smithi151.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-21T08:37:29.639 INFO:teuthology.orchestra.run.smithi028.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-21T08:37:29.656 INFO:teuthology.orchestra.run.smithi028.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2231-g2edfee68-1jammy [22.8 MB] 2024-03-21T08:37:29.749 INFO:teuthology.orchestra.run.smithi151.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-21T08:37:29.760 INFO:teuthology.orchestra.run.smithi151.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2231-g2edfee68-1jammy [5,084 kB] 2024-03-21T08:37:29.876 INFO:teuthology.orchestra.run.smithi151.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-21T08:37:29.902 INFO:teuthology.orchestra.run.smithi028.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-21T08:37:29.919 INFO:teuthology.orchestra.run.smithi151.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-21T08:37:30.008 INFO:teuthology.orchestra.run.smithi151.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-21T08:37:30.017 INFO:teuthology.orchestra.run.smithi151.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-21T08:37:30.030 INFO:teuthology.orchestra.run.smithi151.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-21T08:37:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-21T08:37:30.072 INFO:teuthology.orchestra.run.smithi151.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-21T08:37:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-21T08:37:30.138 INFO:teuthology.orchestra.run.smithi151.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2231-g2edfee68-1jammy [244 kB] 2024-03-21T08:37:30.159 INFO:teuthology.orchestra.run.smithi151.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2231-g2edfee68-1jammy [124 kB] 2024-03-21T08:37:30.162 INFO:teuthology.orchestra.run.smithi028.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-21T08:37:30.170 INFO:teuthology.orchestra.run.smithi151.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2231-g2edfee68-1jammy [1,525 kB] 2024-03-21T08:37:30.172 INFO:teuthology.orchestra.run.smithi028.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-21T08:37:30.184 INFO:teuthology.orchestra.run.smithi028.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-21T08:37:30.226 INFO:teuthology.orchestra.run.smithi028.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-21T08:37:30.285 INFO:teuthology.orchestra.run.smithi151.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2231-g2edfee68-1jammy [6,253 kB] 2024-03-21T08:37:30.336 INFO:teuthology.orchestra.run.smithi151.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-21T08:37:30.337 INFO:teuthology.orchestra.run.smithi151.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-21T08:37:30.339 INFO:teuthology.orchestra.run.smithi151.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-21T08:37:30.490 INFO:teuthology.orchestra.run.smithi028.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-21T08:37:30.491 INFO:teuthology.orchestra.run.smithi028.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-21T08:37:30.492 INFO:teuthology.orchestra.run.smithi028.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-21T08:37:30.506 INFO:teuthology.orchestra.run.smithi151.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-21T08:37:30.656 INFO:teuthology.orchestra.run.smithi028.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-21T08:37:30.769 INFO:teuthology.orchestra.run.smithi151.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2231-g2edfee68-1jammy [22.8 MB] 2024-03-21T08:37:30.856 INFO:teuthology.orchestra.run.smithi151.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-21T08:37:30.953 INFO:teuthology.orchestra.run.smithi028.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2231-g2edfee68-1jammy [14.1 kB] 2024-03-21T08:37:30.953 INFO:teuthology.orchestra.run.smithi028.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2231-g2edfee68-1jammy [870 kB] 2024-03-21T08:37:30.995 INFO:teuthology.orchestra.run.smithi028.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2231-g2edfee68-1jammy [2,387 kB] 2024-03-21T08:37:30.997 INFO:teuthology.orchestra.run.smithi028.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-21T08:37:31.134 INFO:teuthology.orchestra.run.smithi028.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2231-g2edfee68-1jammy [779 kB] 2024-03-21T08:37:31.178 INFO:teuthology.orchestra.run.smithi028.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2231-g2edfee68-1jammy [146 kB] 2024-03-21T08:37:31.187 INFO:teuthology.orchestra.run.smithi028.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2231-g2edfee68-1jammy [1,936 kB] 2024-03-21T08:37:31.263 INFO:teuthology.orchestra.run.smithi151.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-21T08:37:31.272 INFO:teuthology.orchestra.run.smithi151.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-21T08:37:31.273 INFO:teuthology.orchestra.run.smithi151.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-21T08:37:31.297 INFO:teuthology.orchestra.run.smithi028.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2231-g2edfee68-1jammy [8,625 kB] 2024-03-21T08:37:31.343 INFO:teuthology.orchestra.run.smithi151.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-21T08:37:31.343 INFO:teuthology.orchestra.run.smithi151.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-21T08:37:31.344 INFO:teuthology.orchestra.run.smithi151.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-21T08:37:31.345 INFO:teuthology.orchestra.run.smithi151.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-21T08:37:31.346 INFO:teuthology.orchestra.run.smithi151.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-21T08:37:31.346 INFO:teuthology.orchestra.run.smithi151.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-21T08:37:31.402 INFO:teuthology.orchestra.run.smithi028.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-21T08:37:31.411 INFO:teuthology.orchestra.run.smithi028.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-21T08:37:31.412 INFO:teuthology.orchestra.run.smithi028.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-21T08:37:31.482 INFO:teuthology.orchestra.run.smithi028.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-21T08:37:31.483 INFO:teuthology.orchestra.run.smithi028.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-21T08:37:31.483 INFO:teuthology.orchestra.run.smithi028.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-21T08:37:31.484 INFO:teuthology.orchestra.run.smithi028.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-21T08:37:31.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-21T08:37:31.486 INFO:teuthology.orchestra.run.smithi028.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-21T08:37:31.510 INFO:teuthology.orchestra.run.smithi151.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-21T08:37:31.651 INFO:teuthology.orchestra.run.smithi028.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-21T08:37:31.724 INFO:teuthology.orchestra.run.smithi151.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-21T08:37:31.792 INFO:teuthology.orchestra.run.smithi028.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2231-g2edfee68-1jammy [14.2 kB] 2024-03-21T08:37:31.792 INFO:teuthology.orchestra.run.smithi028.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2231-g2edfee68-1jammy [50.2 MB] 2024-03-21T08:37:31.854 INFO:teuthology.orchestra.run.smithi028.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-21T08:37:32.052 INFO:teuthology.orchestra.run.smithi151.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-21T08:37:32.099 INFO:teuthology.orchestra.run.smithi151.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-21T08:37:32.126 INFO:teuthology.orchestra.run.smithi151.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-21T08:37:32.135 INFO:teuthology.orchestra.run.smithi151.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-21T08:37:32.176 INFO:teuthology.orchestra.run.smithi028.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-21T08:37:32.182 INFO:teuthology.orchestra.run.smithi151.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-21T08:37:32.206 INFO:teuthology.orchestra.run.smithi151.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-21T08:37:32.208 INFO:teuthology.orchestra.run.smithi151.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-21T08:37:32.215 INFO:teuthology.orchestra.run.smithi151.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-21T08:37:32.218 INFO:teuthology.orchestra.run.smithi151.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-21T08:37:32.222 INFO:teuthology.orchestra.run.smithi028.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-21T08:37:32.249 INFO:teuthology.orchestra.run.smithi028.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-21T08:37:32.258 INFO:teuthology.orchestra.run.smithi028.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-21T08:37:32.330 INFO:teuthology.orchestra.run.smithi028.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-21T08:37:32.332 INFO:teuthology.orchestra.run.smithi028.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-21T08:37:32.338 INFO:teuthology.orchestra.run.smithi028.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-21T08:37:32.341 INFO:teuthology.orchestra.run.smithi028.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-21T08:37:32.414 INFO:teuthology.orchestra.run.smithi151.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-21T08:37:32.538 INFO:teuthology.orchestra.run.smithi028.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-21T08:37:32.602 INFO:teuthology.orchestra.run.smithi151.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2231-g2edfee68-1jammy [14.1 kB] 2024-03-21T08:37:32.603 INFO:teuthology.orchestra.run.smithi151.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2231-g2edfee68-1jammy [870 kB] 2024-03-21T08:37:32.672 INFO:teuthology.orchestra.run.smithi151.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-21T08:37:32.681 INFO:teuthology.orchestra.run.smithi151.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2231-g2edfee68-1jammy [2,387 kB] 2024-03-21T08:37:32.797 INFO:teuthology.orchestra.run.smithi028.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-21T08:37:32.923 INFO:teuthology.orchestra.run.smithi151.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2231-g2edfee68-1jammy [779 kB] 2024-03-21T08:37:32.934 INFO:teuthology.orchestra.run.smithi151.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-21T08:37:32.936 INFO:teuthology.orchestra.run.smithi151.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-21T08:37:32.952 INFO:teuthology.orchestra.run.smithi151.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-21T08:37:32.957 INFO:teuthology.orchestra.run.smithi151.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-21T08:37:32.995 INFO:teuthology.orchestra.run.smithi151.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-21T08:37:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2231-g2edfee68-1jammy [146 kB] 2024-03-21T08:37:33.009 INFO:teuthology.orchestra.run.smithi151.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-21T08:37:33.010 INFO:teuthology.orchestra.run.smithi151.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2231-g2edfee68-1jammy [1,936 kB] 2024-03-21T08:37:33.060 INFO:teuthology.orchestra.run.smithi028.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-21T08:37:33.061 INFO:teuthology.orchestra.run.smithi028.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-21T08:37:33.077 INFO:teuthology.orchestra.run.smithi028.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-21T08:37:33.082 INFO:teuthology.orchestra.run.smithi028.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-21T08:37:33.100 INFO:teuthology.orchestra.run.smithi151.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-21T08:37:33.102 INFO:teuthology.orchestra.run.smithi151.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-21T08:37:33.103 INFO:teuthology.orchestra.run.smithi151.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-21T08:37:33.120 INFO:teuthology.orchestra.run.smithi028.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-21T08:37:33.134 INFO:teuthology.orchestra.run.smithi028.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-21T08:37:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2231-g2edfee68-1jammy [8,625 kB] 2024-03-21T08:37:33.226 INFO:teuthology.orchestra.run.smithi028.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-21T08:37:33.227 INFO:teuthology.orchestra.run.smithi028.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-21T08:37:33.229 INFO:teuthology.orchestra.run.smithi028.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-21T08:37:33.283 INFO:teuthology.orchestra.run.smithi151.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-21T08:37:33.409 INFO:teuthology.orchestra.run.smithi028.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-21T08:37:33.556 INFO:teuthology.orchestra.run.smithi151.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-21T08:37:33.683 INFO:teuthology.orchestra.run.smithi028.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-21T08:37:33.989 INFO:teuthology.orchestra.run.smithi151.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2231-g2edfee68-1jammy [14.2 kB] 2024-03-21T08:37:33.989 INFO:teuthology.orchestra.run.smithi151.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2231-g2edfee68-1jammy [50.2 MB] 2024-03-21T08:37:34.387 INFO:teuthology.orchestra.run.smithi028.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2231-g2edfee68-1jammy [121 kB] 2024-03-21T08:37:34.388 INFO:teuthology.orchestra.run.smithi028.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2231-g2edfee68-1jammy [228 kB] 2024-03-21T08:37:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2231-g2edfee68-1jammy [34.0 kB] 2024-03-21T08:37:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2231-g2edfee68-1jammy [23.4 kB] 2024-03-21T08:37:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2231-g2edfee68-1jammy [38.5 kB] 2024-03-21T08:37:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2231-g2edfee68-1jammy [13.2 MB] 2024-03-21T08:37:35.082 INFO:teuthology.orchestra.run.smithi028.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2231-g2edfee68-1jammy [91.7 kB] 2024-03-21T08:37:35.752 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 178 MB in 9s (20.3 MB/s) 2024-03-21T08:37:35.893 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-21T08:37:35.962 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-21T08:37:35.965 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-21T08:37:36.008 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-21T08:37:36.335 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-21T08:37:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-21T08:37:36.375 INFO:teuthology.orchestra.run.smithi151.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2231-g2edfee68-1jammy [121 kB] 2024-03-21T08:37:36.376 INFO:teuthology.orchestra.run.smithi151.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2231-g2edfee68-1jammy [228 kB] 2024-03-21T08:37:36.378 INFO:teuthology.orchestra.run.smithi151.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2231-g2edfee68-1jammy [34.0 kB] 2024-03-21T08:37:36.379 INFO:teuthology.orchestra.run.smithi151.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2231-g2edfee68-1jammy [23.4 kB] 2024-03-21T08:37:36.379 INFO:teuthology.orchestra.run.smithi151.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2231-g2edfee68-1jammy [38.5 kB] 2024-03-21T08:37:36.380 INFO:teuthology.orchestra.run.smithi151.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2231-g2edfee68-1jammy [13.2 MB] 2024-03-21T08:37:36.394 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-21T08:37:36.705 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cmd2. 2024-03-21T08:37:36.718 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-21T08:37:36.763 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-21T08:37:36.812 INFO:teuthology.orchestra.run.smithi151.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240320.053002/2edfee68ad7243bed422a049b6f5f58619300f11/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2231-g2edfee68-1jammy [91.7 kB] 2024-03-21T08:37:37.074 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-21T08:37:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-21T08:37:37.158 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-21T08:37:37.419 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-21T08:37:37.434 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-21T08:37:37.444 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 178 MB in 11s (16.8 MB/s) 2024-03-21T08:37:37.469 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-21T08:37:37.590 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-21T08:37:37.659 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-21T08:37:37.663 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-21T08:37:37.713 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-21T08:37:37.838 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-21T08:37:37.853 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-21T08:37:37.897 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-21T08:37:38.217 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-21T08:37:38.231 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-21T08:37:38.275 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-21T08:37:38.554 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cmd2. 2024-03-21T08:37:38.568 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-21T08:37:38.575 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-21T08:37:38.590 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:38.612 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-21T08:37:38.642 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:39.016 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-21T08:37:39.030 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-21T08:37:39.116 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-21T08:37:39.237 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-21T08:37:39.252 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:39.303 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:39.429 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-21T08:37:39.443 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-21T08:37:39.479 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-21T08:37:39.706 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-21T08:37:39.721 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:39.772 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:39.858 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-21T08:37:39.872 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-21T08:37:39.916 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-21T08:37:40.134 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-21T08:37:40.149 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-21T08:37:40.200 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-21T08:37:40.328 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-21T08:37:40.343 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:40.386 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:40.553 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librados2. 2024-03-21T08:37:40.568 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../010-librados2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:40.619 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librados2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:40.858 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-21T08:37:40.872 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:40.908 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:41.106 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librbd1. 2024-03-21T08:37:41.121 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../011-librbd1_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:41.172 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librbd1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:41.261 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-21T08:37:41.276 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-21T08:37:41.344 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:37:41.692 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs2. 2024-03-21T08:37:41.706 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-21T08:37:41.707 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:41.722 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-21T08:37:41.758 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:41.765 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-21T08:37:42.068 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librados2. 2024-03-21T08:37:42.077 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rados. 2024-03-21T08:37:42.083 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../010-librados2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:42.135 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rados (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:42.143 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librados2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:42.471 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-21T08:37:42.486 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:37:42.538 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-argparse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:42.572 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librbd1. 2024-03-21T08:37:42.587 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../011-librbd1_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:42.630 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librbd1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:42.849 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cephfs. 2024-03-21T08:37:42.864 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:42.915 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cephfs (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:43.217 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs2. 2024-03-21T08:37:43.232 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:43.275 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-21T08:37:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:37:43.384 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:43.562 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rados. 2024-03-21T08:37:43.577 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:43.612 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rados (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:43.787 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-prettytable. 2024-03-21T08:37:43.802 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-21T08:37:43.854 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-21T08:37:43.940 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-21T08:37:43.955 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:37:43.998 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-argparse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:44.140 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rbd. 2024-03-21T08:37:44.155 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:44.198 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rbd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:44.218 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cephfs. 2024-03-21T08:37:44.233 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:44.268 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cephfs (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:44.588 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-21T08:37:44.603 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:37:44.618 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-21T08:37:44.633 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-21T08:37:44.646 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:44.684 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-21T08:37:44.959 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-prettytable. 2024-03-21T08:37:44.974 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-21T08:37:45.003 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-21T08:37:45.009 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-21T08:37:45.019 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-21T08:37:45.069 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-21T08:37:45.320 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rbd. 2024-03-21T08:37:45.335 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:45.379 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rbd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-21T08:37:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-21T08:37:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-21T08:37:45.734 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-21T08:37:45.747 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-21T08:37:45.791 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-21T08:37:45.992 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua5.1. 2024-03-21T08:37:46.009 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-21T08:37:46.050 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-21T08:37:46.077 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-21T08:37:46.092 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-21T08:37:46.127 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-21T08:37:46.420 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-any. 2024-03-21T08:37:46.422 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-21T08:37:46.435 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-21T08:37:46.437 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-21T08:37:46.480 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-21T08:37:46.486 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-21T08:37:46.781 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package zip. 2024-03-21T08:37:46.796 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-21T08:37:46.843 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package lua5.1. 2024-03-21T08:37:46.847 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking zip (3.0-12build2) ... 2024-03-21T08:37:46.858 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-21T08:37:46.902 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-21T08:37:47.189 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package lua-any. 2024-03-21T08:37:47.203 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-21T08:37:47.226 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package unzip. 2024-03-21T08:37:47.241 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-21T08:37:47.247 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-21T08:37:47.292 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-21T08:37:47.500 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package zip. 2024-03-21T08:37:47.515 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-21T08:37:47.584 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking zip (3.0-12build2) ... 2024-03-21T08:37:47.696 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package luarocks. 2024-03-21T08:37:47.711 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-21T08:37:47.754 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-21T08:37:47.946 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package unzip. 2024-03-21T08:37:47.961 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-21T08:37:48.013 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-21T08:37:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librgw2. 2024-03-21T08:37:48.202 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../027-librgw2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:48.246 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librgw2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:48.717 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package luarocks. 2024-03-21T08:37:48.732 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-21T08:37:48.776 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-21T08:37:48.974 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rgw. 2024-03-21T08:37:48.989 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:49.032 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:49.227 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librgw2. 2024-03-21T08:37:49.241 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../027-librgw2_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:49.285 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librgw2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:49.352 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-21T08:37:49.367 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-21T08:37:49.410 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-21T08:37:49.688 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libradosstriper1. 2024-03-21T08:37:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:49.737 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libradosstriper1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:49.855 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rgw. 2024-03-21T08:37:49.869 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:49.922 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:50.166 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-common. 2024-03-21T08:37:50.181 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:50.224 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:50.317 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-21T08:37:50.330 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-21T08:37:50.384 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-21T08:37:50.754 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libradosstriper1. 2024-03-21T08:37:50.768 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:50.821 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libradosstriper1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:51.349 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-common. 2024-03-21T08:37:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-base. 2024-03-21T08:37:51.364 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:51.369 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-base (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:51.424 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:52.041 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-21T08:37:52.056 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-21T08:37:52.091 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-21T08:37:52.352 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cheroot. 2024-03-21T08:37:52.367 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-21T08:37:52.410 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-21T08:37:52.564 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-base. 2024-03-21T08:37:52.580 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:37:52.632 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-base (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:37:52.697 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-21T08:37:52.712 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-21T08:37:52.747 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-21T08:37:53.016 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-21T08:37:53.032 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-21T08:37:53.074 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-21T08:37:53.379 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-21T08:37:53.385 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-21T08:37:53.393 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-21T08:37:53.401 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-21T08:37:53.437 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-21T08:37:53.443 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-21T08:37:53.729 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-tempora. 2024-03-21T08:37:53.745 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-21T08:37:53.765 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cheroot. 2024-03-21T08:37:53.780 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-21T08:37:53.788 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-21T08:37:53.815 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-21T08:37:54.049 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-portend. 2024-03-21T08:37:54.064 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-21T08:37:54.099 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-21T08:37:54.119 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-21T08:37:54.134 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-21T08:37:54.169 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-21T08:37:54.402 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-21T08:37:54.418 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-21T08:37:54.460 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-21T08:37:54.481 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-21T08:37:54.496 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-21T08:37:54.556 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-21T08:37:54.746 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-21T08:37:54.762 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-21T08:37:54.804 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-21T08:37:54.860 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-21T08:37:54.875 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-21T08:37:54.918 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-21T08:37:55.107 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-natsort. 2024-03-21T08:37:55.123 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-21T08:37:55.165 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-21T08:37:55.205 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-tempora. 2024-03-21T08:37:55.220 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-21T08:37:55.287 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-21T08:37:55.477 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-logutils. 2024-03-21T08:37:55.492 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-21T08:37:55.535 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-21T08:37:55.591 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-portend. 2024-03-21T08:37:55.606 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-21T08:37:55.650 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-21T08:37:55.871 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-21T08:37:55.887 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-21T08:37:55.912 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-21T08:37:55.927 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-21T08:37:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-21T08:37:55.962 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-21T08:37:56.216 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-mako. 2024-03-21T08:37:56.231 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-21T08:37:56.257 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-21T08:37:56.265 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-21T08:37:56.272 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-21T08:37:56.315 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-21T08:37:56.567 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-21T08:37:56.602 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-21T08:37:56.678 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-natsort. 2024-03-21T08:37:56.693 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-21T08:37:56.736 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-21T08:37:56.896 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-21T08:37:56.911 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-21T08:37:56.988 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-21T08:37:57.023 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-logutils. 2024-03-21T08:37:57.038 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-21T08:37:57.073 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-21T08:37:57.274 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webob. 2024-03-21T08:37:57.289 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-21T08:37:57.332 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-21T08:37:57.402 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-21T08:37:57.417 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-21T08:37:57.460 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-21T08:37:57.693 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-waitress. 2024-03-21T08:37:57.709 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-21T08:37:57.754 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-21T08:37:57.780 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-mako. 2024-03-21T08:37:57.795 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-21T08:37:57.839 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-21T08:37:58.046 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-tempita. 2024-03-21T08:37:58.062 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-21T08:37:58.095 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-21T08:37:58.142 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-21T08:37:58.157 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-21T08:37:58.201 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-21T08:37:58.513 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-21T08:37:58.528 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-21T08:37:58.571 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-21T08:37:58.874 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webob. 2024-03-21T08:37:58.889 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-21T08:37:58.933 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-21T08:37:59.282 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-paste. 2024-03-21T08:37:59.298 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-21T08:37:59.312 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-waitress. 2024-03-21T08:37:59.328 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-21T08:37:59.340 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-21T08:37:59.660 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-21T08:37:59.675 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-21T08:37:59.710 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-21T08:38:00.013 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-21T08:38:00.028 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-21T08:38:00.062 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-21T08:38:00.123 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-21T08:38:00.324 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webtest. 2024-03-21T08:38:00.339 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-21T08:38:00.373 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-21T08:38:00.399 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-tempita. 2024-03-21T08:38:00.414 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-21T08:38:00.457 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-21T08:38:00.668 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pecan. 2024-03-21T08:38:00.684 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-21T08:38:00.726 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-21T08:38:00.753 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-paste. 2024-03-21T08:38:00.768 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-21T08:38:00.811 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-21T08:38:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-certifi. 2024-03-21T08:38:01.054 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-21T08:38:01.089 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-21T08:38:01.131 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-21T08:38:01.146 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-21T08:38:01.181 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-21T08:38:01.425 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-idna. 2024-03-21T08:38:01.440 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-21T08:38:01.483 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-21T08:38:01.485 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-21T08:38:01.500 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-21T08:38:01.543 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-21T08:38:01.811 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-urllib3. 2024-03-21T08:38:01.827 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-21T08:38:01.864 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webtest. 2024-03-21T08:38:01.869 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-21T08:38:01.879 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-21T08:38:01.922 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-21T08:38:02.189 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-requests. 2024-03-21T08:38:02.200 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pecan. 2024-03-21T08:38:02.204 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-21T08:38:02.215 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-21T08:38:02.247 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-21T08:38:02.250 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-21T08:38:02.558 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-21T08:38:02.574 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-21T08:38:02.590 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-certifi. 2024-03-21T08:38:02.605 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-21T08:38:02.633 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-21T08:38:02.648 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-21T08:38:02.945 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-21T08:38:02.960 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:02.985 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-idna. 2024-03-21T08:38:03.000 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-21T08:38:03.002 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:03.043 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-21T08:38:03.365 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-21T08:38:03.380 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:03.397 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-urllib3. 2024-03-21T08:38:03.412 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-21T08:38:03.423 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:03.455 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-21T08:38:03.726 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-21T08:38:03.740 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-21T08:38:03.784 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-21T08:38:03.943 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-requests. 2024-03-21T08:38:03.957 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-21T08:38:04.035 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-21T08:38:04.087 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-openssl. 2024-03-21T08:38:04.102 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-21T08:38:04.137 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-21T08:38:04.338 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-21T08:38:04.353 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-21T08:38:04.396 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-21T08:38:04.440 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr. 2024-03-21T08:38:04.456 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:04.490 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:04.683 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-21T08:38:04.699 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:04.741 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:04.818 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mon. 2024-03-21T08:38:04.834 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:04.867 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mon (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:05.097 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-21T08:38:05.112 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:05.156 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:05.413 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-osd. 2024-03-21T08:38:05.428 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-osd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:05.476 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-21T08:38:05.491 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-21T08:38:05.534 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-21T08:38:05.890 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-openssl. 2024-03-21T08:38:05.903 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-21T08:38:05.946 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-21T08:38:06.258 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph. 2024-03-21T08:38:06.258 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr. 2024-03-21T08:38:06.273 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../068-ceph_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:06.273 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:06.315 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:06.316 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-fuse. 2024-03-21T08:38:06.633 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:06.676 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:06.721 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mon. 2024-03-21T08:38:06.735 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:06.779 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mon (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:07.046 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mds. 2024-03-21T08:38:07.062 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:07.104 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mds (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:07.416 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-osd. 2024-03-21T08:38:07.423 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephadm. 2024-03-21T08:38:07.431 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:07.439 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../071-cephadm_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:07.473 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:07.474 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-osd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:07.793 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-21T08:38:07.808 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-21T08:38:07.851 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-21T08:38:08.154 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-21T08:38:08.169 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-21T08:38:08.212 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:08.312 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph. 2024-03-21T08:38:08.327 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../068-ceph_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:08.370 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:08.632 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-fuse. 2024-03-21T08:38:08.647 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:08.690 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:09.085 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mds. 2024-03-21T08:38:09.100 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:09.135 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mds (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-babel. 2024-03-21T08:38:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-21T08:38:09.224 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:09.530 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephadm. 2024-03-21T08:38:09.546 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../071-cephadm_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:09.589 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jinja2. 2024-03-21T08:38:09.742 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-21T08:38:09.786 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-21T08:38:09.909 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-21T08:38:09.921 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-21T08:38:09.959 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-21T08:38:10.114 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-21T08:38:10.128 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:10.163 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:10.306 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-21T08:38:10.321 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-21T08:38:10.356 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:10.541 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-21T08:38:10.556 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-21T08:38:10.599 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-21T08:38:10.886 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-routes. 2024-03-21T08:38:10.901 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-21T08:38:10.936 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-21T08:38:11.222 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-21T08:38:11.238 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:11.272 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:11.304 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-babel. 2024-03-21T08:38:11.319 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-21T08:38:11.361 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:11.690 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jinja2. 2024-03-21T08:38:11.691 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-21T08:38:11.703 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-21T08:38:11.707 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-21T08:38:11.740 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-21T08:38:11.750 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-21T08:38:12.127 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-21T08:38:12.142 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:12.186 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:12.295 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-joblib. 2024-03-21T08:38:12.311 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-21T08:38:12.355 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-21T08:38:12.515 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-21T08:38:12.532 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-21T08:38:12.573 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-21T08:38:12.698 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-21T08:38:12.714 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-21T08:38:12.850 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-21T08:38:12.885 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-routes. 2024-03-21T08:38:12.900 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-21T08:38:12.943 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-21T08:38:13.144 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn. 2024-03-21T08:38:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-21T08:38:13.203 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-21T08:38:13.289 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-21T08:38:13.304 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:13.347 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:13.801 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-21T08:38:13.816 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-21T08:38:13.851 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-21T08:38:13.928 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-21T08:38:13.942 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:13.986 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:14.522 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-joblib. 2024-03-21T08:38:14.538 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-21T08:38:14.580 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-21T08:38:14.935 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-21T08:38:14.949 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-21T08:38:14.990 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cachetools. 2024-03-21T08:38:15.005 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-21T08:38:15.039 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-21T08:38:15.104 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-21T08:38:15.326 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-21T08:38:15.341 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-21T08:38:15.383 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-21T08:38:15.416 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn. 2024-03-21T08:38:15.431 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-21T08:38:15.474 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-21T08:38:15.720 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-21T08:38:15.737 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-21T08:38:15.778 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-21T08:38:16.073 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rsa. 2024-03-21T08:38:16.089 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-21T08:38:16.131 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-21T08:38:16.194 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-21T08:38:16.208 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:16.252 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:16.468 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-google-auth. 2024-03-21T08:38:16.485 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-21T08:38:16.526 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-21T08:38:16.796 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-21T08:38:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-21T08:38:16.846 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-21T08:38:17.140 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-websocket. 2024-03-21T08:38:17.157 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-21T08:38:17.198 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-21T08:38:17.248 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cachetools. 2024-03-21T08:38:17.263 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-21T08:38:17.306 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-21T08:38:17.510 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-21T08:38:17.526 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-21T08:38:17.568 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-21T08:38:17.626 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-21T08:38:17.642 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-21T08:38:17.684 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-21T08:38:18.039 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-21T08:38:18.055 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-21T08:38:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-21T08:38:18.215 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-21T08:38:18.230 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:18.265 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:18.434 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rsa. 2024-03-21T08:38:18.449 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-21T08:38:18.494 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-21T08:38:18.551 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-21T08:38:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-21T08:38:18.609 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-21T08:38:18.797 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-google-auth. 2024-03-21T08:38:18.812 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-21T08:38:18.863 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-21T08:38:18.954 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-21T08:38:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-21T08:38:19.012 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-21T08:38:19.192 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-21T08:38:19.208 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-21T08:38:19.242 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-21T08:38:19.298 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package jq. 2024-03-21T08:38:19.314 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-21T08:38:19.347 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-21T08:38:19.529 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-websocket. 2024-03-21T08:38:19.545 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-21T08:38:19.587 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-21T08:38:19.883 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package socat. 2024-03-21T08:38:19.897 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-21T08:38:19.941 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-21T08:38:19.942 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-21T08:38:19.957 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-21T08:38:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-21T08:38:20.286 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package xmlstarlet. 2024-03-21T08:38:20.301 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-21T08:38:20.353 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-21T08:38:20.640 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-21T08:38:20.650 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:20.715 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:20.724 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-test. 2024-03-21T08:38:20.737 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:20.782 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-test (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:21.002 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-21T08:38:21.014 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-21T08:38:21.043 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-21T08:38:21.389 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-21T08:38:21.399 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-21T08:38:21.461 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-21T08:38:21.725 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package jq. 2024-03-21T08:38:21.736 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-21T08:38:21.775 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-21T08:38:22.012 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package socat. 2024-03-21T08:38:22.023 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-21T08:38:22.062 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-21T08:38:22.408 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package xmlstarlet. 2024-03-21T08:38:22.419 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-21T08:38:22.457 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-21T08:38:22.787 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-test. 2024-03-21T08:38:22.798 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:22.827 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-volume. 2024-03-21T08:38:22.829 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-test (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:22.842 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:22.894 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-volume (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:23.315 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-21T08:38:23.329 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:23.373 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-mirror (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:23.684 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-colorama. 2024-03-21T08:38:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-21T08:38:23.734 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-21T08:38:23.970 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-shell. 2024-03-21T08:38:23.984 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:24.020 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-shell (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:24.283 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-top. 2024-03-21T08:38:24.297 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:24.340 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-top (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:24.593 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-21T08:38:24.610 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:24.676 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs-dev (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:24.987 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-21T08:38:25.004 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-21T08:38:25.037 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-21T08:38:25.143 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-volume. 2024-03-21T08:38:25.155 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:25.192 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-volume (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:25.383 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-21T08:38:25.398 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-21T08:38:25.441 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-21T08:38:25.656 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-21T08:38:25.668 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:25.706 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephfs-mirror (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:25.820 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package nvme-cli. 2024-03-21T08:38:25.835 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-21T08:38:25.878 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-21T08:38:26.070 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-colorama. 2024-03-21T08:38:26.080 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-21T08:38:26.126 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-21T08:38:26.309 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package pkg-config. 2024-03-21T08:38:26.324 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-21T08:38:26.359 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-21T08:38:26.463 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephfs-shell. 2024-03-21T08:38:26.475 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:26.521 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephfs-shell (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:26.586 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-21T08:38:26.601 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-21T08:38:26.645 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-21T08:38:26.833 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephfs-top. 2024-03-21T08:38:26.845 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2231-g2edfee68-1jammy_all.deb ... 2024-03-21T08:38:26.891 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephfs-top (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:27.056 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-21T08:38:27.072 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-21T08:38:27.114 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-21T08:38:27.177 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-21T08:38:27.190 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:27.236 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs-dev (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:27.442 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastescript. 2024-03-21T08:38:27.460 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-21T08:38:27.502 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-21T08:38:27.614 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-21T08:38:27.626 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-21T08:38:27.681 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-21T08:38:27.820 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pluggy. 2024-03-21T08:38:27.837 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-21T08:38:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-21T08:38:28.111 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-21T08:38:28.123 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-21T08:38:28.169 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-21T08:38:28.173 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-psutil. 2024-03-21T08:38:28.190 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-21T08:38:28.232 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-21T08:38:28.500 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package nvme-cli. 2024-03-21T08:38:28.512 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-21T08:38:28.549 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-21T08:38:28.584 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-py. 2024-03-21T08:38:28.601 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-21T08:38:28.642 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-21T08:38:28.971 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pygments. 2024-03-21T08:38:28.980 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package pkg-config. 2024-03-21T08:38:28.987 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-21T08:38:28.993 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-21T08:38:29.029 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-21T08:38:29.039 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-21T08:38:29.301 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-21T08:38:29.314 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-21T08:38:29.351 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-21T08:38:29.492 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-21T08:38:29.503 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-21T08:38:29.543 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-21T08:38:29.807 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-21T08:38:29.819 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-21T08:38:29.861 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-toml. 2024-03-21T08:38:29.863 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-21T08:38:29.873 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-21T08:38:29.911 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-21T08:38:30.167 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastescript. 2024-03-21T08:38:30.179 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-21T08:38:30.216 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-21T08:38:30.238 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pytest. 2024-03-21T08:38:30.251 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-21T08:38:30.288 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-21T08:38:30.571 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pluggy. 2024-03-21T08:38:30.583 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-21T08:38:30.617 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplejson. 2024-03-21T08:38:30.621 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-21T08:38:30.632 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-21T08:38:30.675 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-21T08:38:30.899 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-psutil. 2024-03-21T08:38:30.911 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-21T08:38:30.941 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-21T08:38:30.945 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-21T08:38:30.958 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-21T08:38:31.003 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-21T08:38:31.294 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-py. 2024-03-21T08:38:31.308 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-21T08:38:31.344 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-21T08:38:31.665 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pygments. 2024-03-21T08:38:31.678 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-21T08:38:31.723 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-21T08:38:31.749 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package radosgw. 2024-03-21T08:38:31.764 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../122-radosgw_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:31.807 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking radosgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:32.162 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-21T08:38:32.174 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-21T08:38:32.203 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-21T08:38:32.502 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package rbd-fuse. 2024-03-21T08:38:32.516 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:32.540 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-toml. 2024-03-21T08:38:32.551 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking rbd-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:32.552 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-21T08:38:32.598 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-21T08:38:32.837 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-21T08:38:32.935 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pytest. 2024-03-21T08:38:32.948 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-21T08:38:32.993 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-21T08:38:33.165 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-21T08:38:33.291 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-21T08:38:33.339 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplejson. 2024-03-21T08:38:33.352 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-21T08:38:33.397 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-21T08:38:33.545 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-21T08:38:33.693 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-21T08:38:33.706 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-21T08:38:33.751 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-21T08:38:33.971 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-21T08:38:34.456 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package radosgw. 2024-03-21T08:38:34.468 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../122-radosgw_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:34.505 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking radosgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:34.621 INFO:teuthology.orchestra.run.smithi028.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-21T08:38:34.634 INFO:teuthology.orchestra.run.smithi028.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-21T08:38:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:35.468 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package rbd-fuse. 2024-03-21T08:38:35.470 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user cephadm....done 2024-03-21T08:38:35.481 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2231-g2edfee68-1jammy_amd64.deb ... 2024-03-21T08:38:35.526 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking rbd-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:35.641 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-21T08:38:35.836 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-21T08:38:35.942 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-21T08:38:36.108 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-21T08:38:36.210 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-21T08:38:36.234 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-21T08:38:36.336 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-21T08:38:36.493 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-21T08:38:36.604 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-21T08:38:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-21T08:38:36.897 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-21T08:38:36.997 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-21T08:38:37.341 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-21T08:38:37.508 INFO:teuthology.orchestra.run.smithi151.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-21T08:38:37.552 INFO:teuthology.orchestra.run.smithi151.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-21T08:38:37.603 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:37.718 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-21T08:38:37.994 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-21T08:38:38.097 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user cephadm....done 2024-03-21T08:38:38.252 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-21T08:38:38.262 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-21T08:38:38.554 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-21T08:38:38.563 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-21T08:38:38.788 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-21T08:38:38.906 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-21T08:38:38.932 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-21T08:38:39.174 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-21T08:38:39.175 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-21T08:38:39.434 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-21T08:38:39.443 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-21T08:38:39.569 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-21T08:38:39.701 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-argparse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:39.888 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-21T08:38:40.019 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-21T08:38:40.145 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-21T08:38:40.232 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-21T08:38:40.271 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-21T08:38:40.531 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-21T08:38:40.534 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-21T08:38:40.632 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-21T08:38:40.835 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-21T08:38:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-21T08:38:41.120 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-21T08:38:41.224 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-21T08:38:41.272 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-21T08:38:41.344 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-21T08:38:41.355 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-21T08:38:41.470 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-21T08:38:41.641 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-21T08:38:41.847 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:41.908 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-21T08:38:41.981 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-21T08:38:42.218 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-argparse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:42.265 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-21T08:38:42.486 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-21T08:38:42.525 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-21T08:38:42.595 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-21T08:38:42.697 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-21T08:38:42.876 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up zip (3.0-12build2) ... 2024-03-21T08:38:42.973 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-21T08:38:43.002 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-21T08:38:43.108 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-21T08:38:43.368 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-21T08:38:43.721 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-21T08:38:43.776 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-21T08:38:43.832 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-21T08:38:43.905 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-21T08:38:43.997 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-21T08:38:44.014 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-21T08:38:44.098 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-21T08:38:44.199 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:44.525 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-21T08:38:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-21T08:38:44.835 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-21T08:38:44.918 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-21T08:38:45.145 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-21T08:38:45.185 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-21T08:38:45.595 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-21T08:38:45.697 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up zip (3.0-12build2) ... 2024-03-21T08:38:45.798 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-21T08:38:46.130 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-21T08:38:46.484 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-21T08:38:46.490 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-21T08:38:46.760 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-21T08:38:46.833 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-21T08:38:46.912 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-21T08:38:47.021 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-21T08:38:47.092 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-21T08:38:47.218 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:47.322 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-21T08:38:47.536 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-idna (3.3-1) ... 2024-03-21T08:38:47.715 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-21T08:38:47.895 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-21T08:38:47.975 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-21T08:38:48.369 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-21T08:38:48.704 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-21T08:38:49.005 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-21T08:38:49.185 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-21T08:38:49.349 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-21T08:38:49.453 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:49.587 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-21T08:38:49.608 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-21T08:38:49.734 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:49.864 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-21T08:38:49.973 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-21T08:38:50.044 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-idna (3.3-1) ... 2024-03-21T08:38:50.082 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-21T08:38:50.379 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-21T08:38:50.392 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-21T08:38:50.669 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-21T08:38:50.936 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:51.062 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-21T08:38:51.388 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-21T08:38:51.588 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-21T08:38:51.665 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-21T08:38:51.814 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:51.924 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-21T08:38:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-21T08:38:52.217 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-21T08:38:52.224 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-21T08:38:52.383 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-21T08:38:52.484 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-21T08:38:52.501 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-21T08:38:52.777 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-21T08:38:52.811 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-21T08:38:52.903 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-21T08:38:53.078 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-21T08:38:53.204 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-21T08:38:53.322 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-21T08:38:53.363 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:53.489 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-21T08:38:53.607 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-21T08:38:53.798 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-21T08:38:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-21T08:38:54.083 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-21T08:38:54.167 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-21T08:38:54.293 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-21T08:38:54.350 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-21T08:38:54.419 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:54.508 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-21T08:38:54.610 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-21T08:38:54.754 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-21T08:38:54.880 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-21T08:38:54.902 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-21T08:38:55.178 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-21T08:38:55.304 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-21T08:38:55.315 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-21T08:38:55.575 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-21T08:38:55.589 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-21T08:38:55.700 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-21T08:38:55.740 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-21T08:38:55.968 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:56.041 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-21T08:38:56.094 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-21T08:38:56.350 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-21T08:38:56.428 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-21T08:38:56.618 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-21T08:38:56.727 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-21T08:38:56.755 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-21T08:38:56.836 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-21T08:38:56.925 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-21T08:38:57.039 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-21T08:38:57.155 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-21T08:38:57.242 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-21T08:38:57.264 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-21T08:38:57.519 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-21T08:38:57.666 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-21T08:38:57.803 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-21T08:38:57.967 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-21T08:38:58.079 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-21T08:38:58.092 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-21T08:38:58.352 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-21T08:38:58.453 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-21T08:38:58.473 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-21T08:38:58.750 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-21T08:38:58.795 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-21T08:38:59.102 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-21T08:38:59.105 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-21T08:38:59.228 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librados2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:59.354 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:59.389 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-21T08:38:59.480 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-21T08:38:59.609 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-21T08:38:59.775 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:38:59.885 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-21T08:38:59.892 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libradosstriper1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:00.052 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-21T08:39:00.145 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-21T08:39:00.362 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librbd1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:00.404 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-21T08:39:00.512 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-modules-core (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:00.622 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:00.794 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-21T08:39:00.832 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-21T08:39:00.832 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-21T08:39:01.116 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-21T08:39:01.465 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs-dev (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:01.468 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-21T08:39:01.583 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rados (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:01.585 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librados2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librgw2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:01.712 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:01.837 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-21T08:39:01.860 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-top (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:02.003 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-21T08:39:02.174 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:02.300 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libradosstriper1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:02.451 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-21T08:39:02.744 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librbd1 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:02.870 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-modules-core (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:02.979 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.173 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-21T08:39:03.173 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-21T08:39:03.356 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rbd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up rbd-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.583 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.709 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cephfs (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.789 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs-dev (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.835 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-shell (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.948 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rados (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:03.961 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:04.073 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librgw2 (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:04.200 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephfs-top (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:04.326 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-21T08:39:04.506 INFO:teuthology.orchestra.run.smithi028.stdout:Adding group ceph....done 2024-03-21T08:39:04.949 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user ceph....done 2024-03-21T08:39:05.109 INFO:teuthology.orchestra.run.smithi028.stdout:Setting system user ceph properties....done 2024-03-21T08:39:05.125 INFO:teuthology.orchestra.run.smithi028.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-21T08:39:05.275 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-21T08:39:05.596 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rbd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:05.658 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-21T08:39:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up rbd-fuse (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:05.822 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:05.973 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cephfs (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:06.099 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephfs-shell (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:06.234 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-common (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:06.396 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-mirror (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:06.630 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-21T08:39:06.630 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-21T08:39:06.778 INFO:teuthology.orchestra.run.smithi151.stdout:Adding group ceph....done 2024-03-21T08:39:07.154 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user ceph....done 2024-03-21T08:39:07.256 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-test (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:07.324 INFO:teuthology.orchestra.run.smithi151.stdout:Setting system user ceph properties....done 2024-03-21T08:39:07.339 INFO:teuthology.orchestra.run.smithi151.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-21T08:39:07.440 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up radosgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:07.482 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-21T08:39:07.852 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-21T08:39:07.998 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-21T08:39:07.999 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-21T08:39:08.501 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephfs-mirror (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:08.634 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-base (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:08.695 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-21T08:39:08.695 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-21T08:39:09.052 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-21T08:39:09.310 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-test (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:09.454 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up radosgw (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:09.752 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mds (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:09.944 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-21T08:39:09.944 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-21T08:39:09.999 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-21T08:39:09.999 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-21T08:39:10.546 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-base (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:10.654 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:10.892 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-21T08:39:10.892 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-21T08:39:10.903 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-21T08:39:11.548 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-osd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:11.549 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mds (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:11.772 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-21T08:39:11.772 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-21T08:39:11.886 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-21T08:39:11.887 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-21T08:39:12.392 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:12.499 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:12.625 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:12.639 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-21T08:39:12.639 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-21T08:39:12.785 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mon (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:13.030 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-21T08:39:13.030 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-21T08:39:13.294 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-osd (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:13.601 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-21T08:39:13.602 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-21T08:39:13.653 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:13.795 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:13.913 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-dashboard (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:14.047 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-volume (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:14.229 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:14.329 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:14.400 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-21T08:39:14.464 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mon (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:14.517 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-21T08:39:14.672 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-21T08:39:14.672 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-21T08:39:14.703 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-21T08:39:15.323 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-cephadm (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:15.466 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:15.625 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-dashboard (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:15.768 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-volume (19.0.0-2231-g2edfee68-1jammy) ... 2024-03-21T08:39:16.113 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-21T08:39:16.255 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-21T08:39:16.538 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-21T08:39:16.691 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-21T08:39:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-21T08:39:16.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-21T08:39:16.980 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2024-03-21T08:39:17.156 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2024-03-21T08:39:17.156 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2024-03-21T08:39:17.319 INFO:teuthology.orchestra.run.smithi028.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-21T08:39:17.319 INFO:teuthology.orchestra.run.smithi028.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-21T08:39:17.319 INFO:teuthology.orchestra.run.smithi028.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-21T08:39:17.320 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-21T08:39:17.321 INFO:teuthology.orchestra.run.smithi028.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-21T08:39:17.321 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-21T08:39:17.321 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2024-03-21T08:39:17.321 INFO:teuthology.orchestra.run.smithi028.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-21T08:39:17.321 INFO:teuthology.orchestra.run.smithi028.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-21T08:39:17.322 INFO:teuthology.orchestra.run.smithi028.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-21T08:39:17.322 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-21T08:39:17.322 INFO:teuthology.orchestra.run.smithi028.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-21T08:39:17.323 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2024-03-21T08:39:17.323 INFO:teuthology.orchestra.run.smithi028.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-21T08:39:17.323 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-21T08:39:17.323 INFO:teuthology.orchestra.run.smithi028.stdout: libengine-pkcs11-openssl 2024-03-21T08:39:17.373 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2024-03-21T08:39:17.374 INFO:teuthology.orchestra.run.smithi028.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-21T08:39:17.374 INFO:teuthology.orchestra.run.smithi028.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-21T08:39:17.374 INFO:teuthology.orchestra.run.smithi028.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-21T08:39:17.375 INFO:teuthology.orchestra.run.smithi028.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-21T08:39:17.375 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-21T08:39:17.375 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-21T08:39:17.375 INFO:teuthology.orchestra.run.smithi028.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-21T08:39:17.375 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2024-03-21T08:39:17.376 INFO:teuthology.orchestra.run.smithi028.stdout: libssl3 2024-03-21T08:39:17.475 INFO:teuthology.orchestra.run.smithi028.stdout:1 upgraded, 39 newly installed, 0 to remove and 287 not upgraded. 2024-03-21T08:39:17.475 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 54.6 MB of archives. 2024-03-21T08:39:17.475 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-21T08:39:17.475 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-21T08:39:17.714 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-21T08:39:17.866 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-21T08:39:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-21T08:39:17.972 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-21T08:39:17.973 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-21T08:39:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-21T08:39:17.977 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-21T08:39:17.977 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-21T08:39:17.977 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-21T08:39:17.978 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-21T08:39:17.978 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-21T08:39:18.000 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-21T08:39:18.022 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-21T08:39:18.560 INFO:teuthology.orchestra.run.smithi151.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-21T08:39:18.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-21T08:39:18.637 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-03-21T08:39:18.809 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2024-03-21T08:39:18.810 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2024-03-21T08:39:18.855 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-21T08:39:18.855 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-21T08:39:18.855 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-21T08:39:18.856 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-21T08:39:18.857 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-21T08:39:18.878 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-21T08:39:18.952 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-21T08:39:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-21T08:39:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-21T08:39:18.966 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages were automatically installed and are no longer required: 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-21T08:39:18.967 INFO:teuthology.orchestra.run.smithi151.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-21T08:39:18.968 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2024-03-21T08:39:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-21T08:39:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-21T08:39:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-21T08:39:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-21T08:39:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-21T08:39:18.970 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2024-03-21T08:39:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-21T08:39:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-21T08:39:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: libengine-pkcs11-openssl 2024-03-21T08:39:18.985 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-21T08:39:18.986 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-21T08:39:19.005 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-21T08:39:19.019 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2024-03-21T08:39:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-21T08:39:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-21T08:39:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-21T08:39:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-21T08:39:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-21T08:39:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-21T08:39:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-21T08:39:19.021 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2024-03-21T08:39:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: libssl3 2024-03-21T08:39:19.087 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-21T08:39:19.094 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-21T08:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-21T08:39:19.095 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-21T08:39:19.096 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-21T08:39:19.116 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-21T08:39:19.118 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-21T08:39:19.206 INFO:teuthology.orchestra.run.smithi151.stdout:1 upgraded, 39 newly installed, 0 to remove and 287 not upgraded. 2024-03-21T08:39:19.206 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 54.6 MB of archives. 2024-03-21T08:39:19.207 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-21T08:39:19.207 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-21T08:39:19.476 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-21T08:39:19.476 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-21T08:39:19.476 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-21T08:39:19.476 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-21T08:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-21T08:39:19.477 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-21T08:39:19.500 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-21T08:39:19.501 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-21T08:39:19.705 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-21T08:39:19.894 INFO:teuthology.orchestra.run.smithi028.stdout:Preconfiguring packages ... 2024-03-21T08:39:19.963 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-21T08:39:19.998 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-21T08:39:20.000 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-21T08:39:20.005 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-21T08:39:20.008 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-21T08:39:20.009 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-21T08:39:20.011 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-21T08:39:20.014 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-21T08:39:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 54.6 MB in 2s (25.8 MB/s) 2024-03-21T08:39:20.179 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-21T08:39:20.179 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-21T08:39:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package flex. 2024-03-21T08:39:20.304 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140587 files and directories currently installed.) 2024-03-21T08:39:20.307 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:20.370 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-21T08:39:20.436 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-21T08:39:20.940 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-21T08:39:21.168 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-21T08:39:21.464 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-21T08:39:21.466 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-21T08:39:21.466 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-21T08:39:21.469 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-21T08:39:21.469 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-21T08:39:21.473 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-21T08:39:21.489 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-21T08:39:21.497 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-21T08:39:21.662 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-21T08:39:21.702 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:22.287 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package bison. 2024-03-21T08:39:22.362 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140660 files and directories currently installed.) 2024-03-21T08:39:22.366 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-21T08:39:22.400 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-21T08:39:22.400 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-21T08:39:22.414 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-21T08:39:22.427 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-21T08:39:22.504 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-21T08:39:22.512 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-21T08:39:22.512 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-21T08:39:22.513 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-21T08:39:22.514 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-21T08:39:22.542 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-21T08:39:22.545 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-21T08:39:22.840 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-21T08:39:22.855 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-21T08:39:22.898 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-21T08:39:23.175 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-21T08:39:23.226 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package dump. 2024-03-21T08:39:23.242 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-21T08:39:23.257 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-21T08:39:23.257 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-21T08:39:23.258 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-21T08:39:23.285 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking dump (0.4b47-3) ... 2024-03-21T08:39:23.351 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-21T08:39:23.365 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-21T08:39:23.580 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package indent. 2024-03-21T08:39:23.596 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:23.638 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-21T08:39:23.817 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-21T08:39:23.818 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-21T08:39:23.932 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-21T08:39:23.949 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-21T08:39:24.208 INFO:teuthology.orchestra.run.smithi151.stdout:Preconfiguring packages ... 2024-03-21T08:39:24.269 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-21T08:39:24.286 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-21T08:39:24.327 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-21T08:39:24.373 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 54.6 MB in 5s (11.4 MB/s) 2024-03-21T08:39:24.515 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package flex. 2024-03-21T08:39:24.586 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140587 files and directories currently installed.) 2024-03-21T08:39:24.589 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:24.596 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-21T08:39:24.613 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-21T08:39:24.639 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-21T08:39:24.655 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-21T08:39:25.000 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-21T08:39:25.017 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:25.058 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:25.233 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-21T08:39:25.302 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-21T08:39:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:25.352 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:25.403 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-21T08:39:25.598 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjson-perl. 2024-03-21T08:39:25.613 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-21T08:39:25.655 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-21T08:39:25.808 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:25.974 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-21T08:39:25.991 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-21T08:39:26.032 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-21T08:39:26.354 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package bison. 2024-03-21T08:39:26.424 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140660 files and directories currently installed.) 2024-03-21T08:39:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-21T08:39:26.506 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-21T08:39:26.940 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-21T08:39:26.954 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-21T08:39:26.998 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-21T08:39:27.359 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package dump. 2024-03-21T08:39:27.374 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-21T08:39:27.409 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking dump (0.4b47-3) ... 2024-03-21T08:39:27.411 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-21T08:39:27.428 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-21T08:39:27.461 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:27.671 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package indent. 2024-03-21T08:39:27.685 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:27.713 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-21T08:39:27.729 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-21T08:39:27.730 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-21T08:39:27.772 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:27.990 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-21T08:39:28.005 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-21T08:39:28.081 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-21T08:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-21T08:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-21T08:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:28.393 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-21T08:39:28.409 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-21T08:39:28.451 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-21T08:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-21T08:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-21T08:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-21T08:39:28.835 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-21T08:39:28.851 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-21T08:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-21T08:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-21T08:39:28.887 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-21T08:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-21T08:39:29.207 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-21T08:39:29.223 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:29.265 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-21T08:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:29.526 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-21T08:39:29.542 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-21T08:39:29.609 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:29.677 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-21T08:39:29.694 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:29.735 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:29.862 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjson-perl. 2024-03-21T08:39:29.877 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-21T08:39:29.920 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-21T08:39:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-21T08:39:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:30.096 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:30.206 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-21T08:39:30.222 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-21T08:39:30.256 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-21T08:39:30.298 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-21T08:39:30.315 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-21T08:39:30.356 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:30.860 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-21T08:39:30.876 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-21T08:39:30.918 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-21T08:39:31.288 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package modemmanager. 2024-03-21T08:39:31.305 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-21T08:39:31.346 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:31.692 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-21T08:39:31.708 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-21T08:39:31.750 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:32.019 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-21T08:39:32.035 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-21T08:39:32.062 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package network-manager. 2024-03-21T08:39:32.077 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:32.078 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-21T08:39:32.128 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-21T08:39:32.447 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-21T08:39:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-21T08:39:32.505 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:32.807 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-21T08:39:32.811 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ppp. 2024-03-21T08:39:32.823 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-21T08:39:32.827 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-21T08:39:32.857 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-21T08:39:33.175 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-21T08:39:33.202 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-21T08:39:33.218 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-21T08:39:33.260 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-21T08:39:33.523 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package pptp-linux. 2024-03-21T08:39:33.538 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-21T08:39:33.581 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-21T08:39:33.621 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-21T08:39:33.637 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:33.696 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:33.826 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-21T08:39:33.841 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-21T08:39:33.876 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-21T08:39:34.082 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-21T08:39:34.098 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:34.131 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:34.170 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-21T08:39:34.186 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-21T08:39:34.220 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client-common (238) ... 2024-03-21T08:39:34.393 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-21T08:39:34.409 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:34.459 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:34.524 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-21T08:39:34.540 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:34.582 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:34.754 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-21T08:39:34.770 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-21T08:39:34.812 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:35.087 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-common. 2024-03-21T08:39:35.103 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-21T08:39:35.156 INFO:teuthology.orchestra.run.smithi028.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-21T08:39:35.186 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-common (238) ... 2024-03-21T08:39:35.357 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-21T08:39:35.372 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-21T08:39:35.415 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-21T08:39:35.669 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-14. 2024-03-21T08:39:35.686 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:35.730 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:35.777 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package modemmanager. 2024-03-21T08:39:35.793 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-21T08:39:35.834 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:36.507 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package network-manager. 2024-03-21T08:39:36.523 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-21T08:39:36.573 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-21T08:39:36.797 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql. 2024-03-21T08:39:36.813 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-21T08:39:36.856 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql (14+238) ... 2024-03-21T08:39:37.155 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ppp. 2024-03-21T08:39:37.170 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-21T08:39:37.506 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-21T08:39:37.706 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client. 2024-03-21T08:39:37.723 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-21T08:39:37.768 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client (14+238) ... 2024-03-21T08:39:37.810 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package pptp-linux. 2024-03-21T08:39:37.824 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-21T08:39:37.860 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-21T08:39:38.004 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-21T08:39:38.020 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-21T08:39:38.062 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-21T08:39:38.070 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-21T08:39:38.085 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-21T08:39:38.120 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-21T08:39:38.356 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package tcl8.6. 2024-03-21T08:39:38.376 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-21T08:39:38.415 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-21T08:39:38.474 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-21T08:39:38.488 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-21T08:39:38.531 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-client-common (238) ... 2024-03-21T08:39:38.692 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package tcl. 2024-03-21T08:39:38.709 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-21T08:39:38.755 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-21T08:39:38.843 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-21T08:39:38.858 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:38.901 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:39.103 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package wpasupplicant. 2024-03-21T08:39:39.120 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-21T08:39:39.161 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-21T08:39:39.531 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-common. 2024-03-21T08:39:39.532 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-21T08:39:39.546 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-21T08:39:39.549 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-21T08:39:39.582 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-21T08:39:39.624 INFO:teuthology.orchestra.run.smithi151.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-21T08:39:39.654 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-common (238) ... 2024-03-21T08:39:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-21T08:39:39.977 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-21T08:39:40.027 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-21T08:39:40.231 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-14. 2024-03-21T08:39:40.245 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-21T08:39:40.283 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:40.322 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client-common (238) ... 2024-03-21T08:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-21T08:39:40.726 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-21T08:39:40.927 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:41.078 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-21T08:39:41.229 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-21T08:39:41.355 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:41.458 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql. 2024-03-21T08:39:41.473 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-21T08:39:41.533 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql (14+238) ... 2024-03-21T08:39:41.777 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-client. 2024-03-21T08:39:41.792 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-21T08:39:41.827 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-client (14+238) ... 2024-03-21T08:39:42.037 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-21T08:39:42.052 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-21T08:39:42.095 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-21T08:39:42.325 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-21T08:39:42.428 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-21T08:39:42.448 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package tcl8.6. 2024-03-21T08:39:42.464 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-21T08:39:42.506 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-21T08:39:42.775 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package tcl. 2024-03-21T08:39:42.791 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-21T08:39:42.838 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-21T08:39:42.856 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-21T08:39:42.857 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-21T08:39:43.180 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package wpasupplicant. 2024-03-21T08:39:43.196 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-21T08:39:43.238 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-21T08:39:43.564 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-21T08:39:43.618 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-21T08:39:43.634 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-21T08:39:43.675 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-21T08:39:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:43.882 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-21T08:39:44.034 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-21T08:39:44.087 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-21T08:39:44.103 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-21T08:39:44.153 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:44.344 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-21T08:39:44.447 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-client-common (238) ... 2024-03-21T08:39:44.512 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-21T08:39:44.668 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-21T08:39:44.794 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-21T08:39:44.929 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:45.080 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-21T08:39:45.206 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-21T08:39:45.276 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-21T08:39:45.332 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up dump (0.4b47-3) ... 2024-03-21T08:39:45.547 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-21T08:39:45.620 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:45.771 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-21T08:39:45.897 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-21T08:39:46.023 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-21T08:39:46.174 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-21T08:39:46.281 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-21T08:39:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-21T08:39:46.376 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-21T08:39:46.404 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-21T08:39:46.527 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-21T08:39:46.737 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-21T08:39:46.783 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-21T08:39:46.783 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-21T08:39:46.888 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:47.022 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-21T08:39:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client (14+238) ... 2024-03-21T08:39:47.450 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-21T08:39:47.482 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-21T08:39:47.608 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-21T08:39:47.742 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-21T08:39:47.851 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-21T08:39:47.969 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:48.095 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-21T08:39:48.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:48.159 INFO:teuthology.orchestra.run.smithi028.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi028.stdout:which means they are currently configured by ifupdown: 2024-03-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi028.stdout:- ens1f0 2024-03-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi028.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi028.stdout:remove their configuration from /etc/network/interfaces. 2024-03-21T08:39:48.160 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:48.221 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-21T08:39:48.302 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-21T08:39:48.693 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-21T08:39:48.943 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-21T08:39:49.044 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up dump (0.4b47-3) ... 2024-03-21T08:39:49.092 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-21T08:39:49.148 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-21T08:39:49.254 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:49.382 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-21T08:39:49.506 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-21T08:39:49.632 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-21T08:39:49.750 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-21T08:39:49.795 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:49.831 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-21T08:39:49.893 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-21T08:39:49.913 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:50.014 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-21T08:39:50.115 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-common (238) ... 2024-03-21T08:39:50.811 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-21T08:39:50.987 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-21T08:39:51.079 INFO:teuthology.orchestra.run.smithi028.stdout:Adding user postgres to group ssl-cert 2024-03-21T08:39:51.138 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:51.264 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-21T08:39:51.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:51.317 INFO:teuthology.orchestra.run.smithi028.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-21T08:39:51.432 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-client (14+238) ... 2024-03-21T08:39:51.550 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-21T08:39:51.578 INFO:teuthology.orchestra.run.smithi028.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-21T08:39:51.578 INFO:teuthology.orchestra.run.smithi028.stdout:Removing obsolete dictionary files: 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout:which means they are currently configured by ifupdown: 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout:- enp3s0f1 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout:remove their configuration from /etc/network/interfaces. 2024-03-21T08:39:52.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:52.223 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-21T08:39:52.264 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-21T08:39:52.613 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-21T08:39:52.742 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:53.015 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-21T08:39:53.098 INFO:teuthology.orchestra.run.smithi028.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-21T08:39:53.121 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:This user must also own the server process. 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:The database cluster will be initialized with locale "en_US". 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:The default text search configuration will be set to "english". 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:Data page checksums are disabled. 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:39:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-21T08:39:53.162 INFO:teuthology.orchestra.run.smithi028.stdout:creating subdirectories ... ok 2024-03-21T08:39:53.162 INFO:teuthology.orchestra.run.smithi028.stdout:selecting dynamic shared memory implementation ... posix 2024-03-21T08:39:53.187 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default max_connections ... 100 2024-03-21T08:39:53.222 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default shared_buffers ... 128MB 2024-03-21T08:39:53.237 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default time zone ... Etc/UTC 2024-03-21T08:39:53.238 INFO:teuthology.orchestra.run.smithi028.stdout:creating configuration files ... ok 2024-03-21T08:39:53.413 INFO:teuthology.orchestra.run.smithi028.stdout:running bootstrap script ... ok 2024-03-21T08:39:53.686 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-21T08:39:53.813 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-21T08:39:53.938 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-21T08:39:54.056 INFO:teuthology.orchestra.run.smithi028.stdout:performing post-bootstrap initialization ... ok 2024-03-21T08:39:54.064 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-common (238) ... 2024-03-21T08:39:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:syncing data to disk ... ok 2024-03-21T08:39:54.973 INFO:teuthology.orchestra.run.smithi151.stdout:Adding user postgres to group ssl-cert 2024-03-21T08:39:55.161 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-21T08:39:55.238 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:55.239 INFO:teuthology.orchestra.run.smithi151.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-21T08:39:55.489 INFO:teuthology.orchestra.run.smithi151.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-21T08:39:55.489 INFO:teuthology.orchestra.run.smithi151.stdout:Removing obsolete dictionary files: 2024-03-21T08:39:56.192 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-21T08:39:56.657 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-21T08:39:56.999 INFO:teuthology.orchestra.run.smithi151.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-21T08:39:57.034 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-21T08:39:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-21T08:39:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:This user must also own the server process. 2024-03-21T08:39:57.083 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:57.083 INFO:teuthology.orchestra.run.smithi151.stdout:The database cluster will be initialized with locale "en_US". 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout:The default text search configuration will be set to "english". 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout:Data page checksums are disabled. 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-21T08:39:57.084 INFO:teuthology.orchestra.run.smithi151.stdout:creating subdirectories ... ok 2024-03-21T08:39:57.085 INFO:teuthology.orchestra.run.smithi151.stdout:selecting dynamic shared memory implementation ... posix 2024-03-21T08:39:57.108 INFO:teuthology.orchestra.run.smithi151.stdout:selecting default max_connections ... 100 2024-03-21T08:39:57.142 INFO:teuthology.orchestra.run.smithi151.stdout:selecting default shared_buffers ... 128MB 2024-03-21T08:39:57.159 INFO:teuthology.orchestra.run.smithi151.stdout:selecting default time zone ... Etc/UTC 2024-03-21T08:39:57.159 INFO:teuthology.orchestra.run.smithi151.stdout:creating configuration files ... ok 2024-03-21T08:39:57.320 INFO:teuthology.orchestra.run.smithi151.stdout:running bootstrap script ... ok 2024-03-21T08:39:57.938 INFO:teuthology.orchestra.run.smithi151.stdout:performing post-bootstrap initialization ... ok 2024-03-21T08:39:58.140 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:58.266 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-21T08:39:58.392 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql (14+238) ... 2024-03-21T08:39:58.527 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:39:58.528 INFO:teuthology.orchestra.run.smithi151.stdout:syncing data to disk ... ok 2024-03-21T08:39:58.636 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:39:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-21T08:39:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-21T08:39:59.040 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-21T08:39:59.647 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-21T08:39:59.965 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-21T08:40:00.134 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-21T08:40:01.645 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-21T08:40:01.737 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-21T08:40:02.097 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:40:02.198 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-21T08:40:02.273 INFO:teuthology.orchestra.run.smithi028.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-21T08:40:02.280 DEBUG:teuthology.parallel:result is None 2024-03-21T08:40:02.316 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up postgresql (14+238) ... 2024-03-21T08:40:02.442 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-21T08:40:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-21T08:40:02.874 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-21T08:40:02.874 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-21T08:40:03.595 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-21T08:40:03.914 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-21T08:40:04.048 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-21T08:40:05.476 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-21T08:40:05.576 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-21T08:40:06.149 INFO:teuthology.orchestra.run.smithi151.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-21T08:40:06.155 DEBUG:teuthology.parallel:result is None 2024-03-21T08:40:06.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T08:40:06.306 DEBUG:teuthology.orchestra.run.smithi028:> dpkg-query -W -f '${Version}' ceph 2024-03-21T08:40:06.330 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:40:06.331 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:40:06.331 INFO:teuthology.task.install:The correct ceph version 19.0.0-2231-g2edfee68-1jammy is installed. 2024-03-21T08:40:06.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2edfee68ad7243bed422a049b6f5f58619300f11 2024-03-21T08:40:06.478 DEBUG:teuthology.orchestra.run.smithi151:> dpkg-query -W -f '${Version}' ceph 2024-03-21T08:40:06.501 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:40:06.501 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2231-g2edfee68-1jammy 2024-03-21T08:40:06.501 INFO:teuthology.task.install:The correct ceph version 19.0.0-2231-g2edfee68-1jammy is installed. 2024-03-21T08:40:06.503 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-21T08:40:06.503 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:06.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T08:40:06.525 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:06.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T08:40:06.561 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-21T08:40:06.561 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:06.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T08:40:06.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T08:40:06.652 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:06.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T08:40:06.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T08:40:06.731 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-21T08:40:06.732 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:06.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T08:40:06.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T08:40:06.817 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:06.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T08:40:06.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T08:40:06.900 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-21T08:40:06.900 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:06.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T08:40:06.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T08:40:06.984 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:06.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T08:40:07.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T08:40:07.064 INFO:teuthology.run_tasks:Running task ceph... 2024-03-21T08:40:07.151 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-21T08:40:07.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-03-21T08:40:07.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2024-03-21T08:40:07.173 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-21T08:40:07.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-21T08:40:07.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-21T08:40:07.238 INFO:tasks.ceph:Creating extra log directories... 2024-03-21T08:40:07.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-21T08:40:07.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-21T08:40:07.331 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-21T08:40:07.331 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-21T08:40:07.331 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-21T08:40:07.331 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:07.331 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-21T08:40:07.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-21T08:40:07.447 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:07.447 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-21T08:40:07.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-21T08:40:07.548 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-21T08:40:07.548 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-21T08:40:07.548 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126', 'branch': 'wip-vshankar-testing-20240320.053002', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7612126', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240320.053002'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240320.053002', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8073, 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'sleep_before_teardown': 0, 'subset': '203/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240320.053002', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqM1DoNJIt2li0uTk/Ne4B6R5ZL8dWRK89+IfGdoWYkOgN7cBCrLUdVrRidmATZeTEUnWuEUd9RvhI3Jf/GsMc=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVQ4EoPkwqcFmwRr8rgNQDfaP1UqmopUAgnGlaKXBO7f187Ba+V0DzAY2sUYrJcKh7VjUHBywJPbUrQgUqXU2w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '2edfee68ad7243bed422a049b6f5f58619300f11', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-20_11:21:18', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-21T08:40:07.549 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-21T08:40:07.552 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-21T08:40:07.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-03-21T08:40:07.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-03-21T08:40:07.623 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:07.623 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T08:40:07.673 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T08:40:07.673 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-03-21T08:40:07.725 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 08:33:37.941431915 +0000 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 08:33:37.657434450 +0000 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 08:33:37.657434450 +0000 2024-03-21T08:40:07.726 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 08:33:37.657434450 +0000 2024-03-21T08:40:07.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T08:40:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T08:40:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T08:40:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000268998 s, 1.9 MB/s 2024-03-21T08:40:07.789 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T08:40:07.847 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-03-21T08:40:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T08:40:07.898 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:07.898 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-03-21T08:40:07.898 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:07.899 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 08:33:38.445427413 +0000 2024-03-21T08:40:07.899 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 08:33:38.169429877 +0000 2024-03-21T08:40:07.899 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 08:33:38.169429877 +0000 2024-03-21T08:40:07.899 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 08:33:38.169429877 +0000 2024-03-21T08:40:07.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T08:40:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T08:40:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T08:40:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000263787 s, 1.9 MB/s 2024-03-21T08:40:07.962 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T08:40:08.019 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 08:33:38.961422804 +0000 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 08:33:38.701425126 +0000 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 08:33:38.701425126 +0000 2024-03-21T08:40:08.070 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 08:33:38.701425126 +0000 2024-03-21T08:40:08.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T08:40:08.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T08:40:08.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T08:40:08.131 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00026276 s, 1.9 MB/s 2024-03-21T08:40:08.133 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T08:40:08.191 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 953 Links: 1 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-21 08:33:39.485418124 +0000 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-21 08:33:39.209420589 +0000 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-21 08:33:39.209420589 +0000 2024-03-21T08:40:08.242 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-21 08:33:39.209420589 +0000 2024-03-21T08:40:08.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T08:40:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-21T08:40:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-21T08:40:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000261964 s, 2.0 MB/s 2024-03-21T08:40:08.306 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T08:40:08.359 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:08.359 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:08.359 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T08:40:08.369 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T08:40:08.369 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-21T08:40:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T08:40:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.418 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-03-21T08:40:08.418 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.419 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-21 08:33:31.376297388 +0000 2024-03-21T08:40:08.419 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-21 08:33:31.372297424 +0000 2024-03-21T08:40:08.419 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-21 08:33:31.372297424 +0000 2024-03-21T08:40:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-21 08:33:31.372297424 +0000 2024-03-21T08:40:08.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T08:40:08.480 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-21T08:40:08.480 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-21T08:40:08.480 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000411572 s, 1.2 MB/s 2024-03-21T08:40:08.482 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T08:40:08.536 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-21T08:40:08.586 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T08:40:08.586 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-03-21T08:40:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-21 08:33:32.160290385 +0000 2024-03-21T08:40:08.587 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-21 08:33:31.916292564 +0000 2024-03-21T08:40:08.587 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-21 08:33:31.916292564 +0000 2024-03-21T08:40:08.587 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-21 08:33:31.916292564 +0000 2024-03-21T08:40:08.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T08:40:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-21T08:40:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-21T08:40:08.653 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000382433 s, 1.3 MB/s 2024-03-21T08:40:08.654 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T08:40:08.712 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-21 08:33:32.648286025 +0000 2024-03-21T08:40:08.762 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-21 08:33:32.392288313 +0000 2024-03-21T08:40:08.763 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-21 08:33:32.392288313 +0000 2024-03-21T08:40:08.763 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-21 08:33:32.392288313 +0000 2024-03-21T08:40:08.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T08:40:08.827 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-21T08:40:08.827 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-21T08:40:08.827 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000378758 s, 1.4 MB/s 2024-03-21T08:40:08.827 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T08:40:08.880 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-21 08:33:33.184281238 +0000 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-21 08:33:32.916283632 +0000 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-21 08:33:32.916283632 +0000 2024-03-21T08:40:08.930 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-21 08:33:32.916283632 +0000 2024-03-21T08:40:08.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T08:40:08.992 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-21T08:40:08.992 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-21T08:40:08.992 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000395072 s, 1.3 MB/s 2024-03-21T08:40:08.994 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T08:40:09.048 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:09.048 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-21T08:40:09.048 INFO:tasks.ceph:Generating config... 2024-03-21T08:40:09.049 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-21T08:40:09.049 INFO:tasks.ceph:[client] debug client = 20 2024-03-21T08:40:09.049 INFO:tasks.ceph:[client] debug ms = 1 2024-03-21T08:40:09.049 INFO:tasks.ceph:[client] pid file = 2024-03-21T08:40:09.050 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-21T08:40:09.050 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-21T08:40:09.050 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-21T08:40:09.051 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-21T08:40:09.051 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-21T08:40:09.051 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-21T08:40:09.052 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-21T08:40:09.052 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-21T08:40:09.052 INFO:tasks.ceph:Setting up mon.a... 2024-03-21T08:40:09.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-21T08:40:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-03-21T08:40:09.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-21T08:40:09.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-21T08:40:09.210 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.151'), ('mon.c', '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]')] 2024-03-21T08:40:09.211 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-21T08:40:09.212 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:09.212 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-21T08:40:09.261 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.151 --addv c '[v2:172.21.15.151:3301,v1:172.21.15.151:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 51dba44a-cba1-4b57-a836-26a407968834 2024-03-21T08:40:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 51dba44a-cba1-4b57-a836-26a407968834 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-03-21T08:40:09.356291+0000 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-03-21T08:40:09.356291+0000 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 17 (quincy) 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.b 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] mon.c 2024-03-21T08:40:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-21T08:40:09.355 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-21T08:40:09.363 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 51dba44a-cba1-4b57-a836-26a407968834... 2024-03-21T08:40:09.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-21T08:40:09.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-03-21T08:40:09.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-03-21T08:40:09.449 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-21T08:40:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]" 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 51dba44a-cba1-4b57-a836-26a407968834 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.451 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-03-21T08:40:09.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-03-21T08:40:09.453 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-03-21T08:40:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-03-21T08:40:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-03-21T08:40:09.456 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-03-21T08:40:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-03-21T08:40:09.458 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-03-21T08:40:09.459 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.460 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-21T08:40:09.461 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-03-21T08:40:09.462 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "172.21.15.28,172.21.15.151,[v2:172.21.15.151:3301,v1:172.21.15.151:6790]" 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 51dba44a-cba1-4b57-a836-26a407968834 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-03-21T08:40:09.463 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug trim objects = true 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.464 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-03-21T08:40:09.465 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 20 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 20 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 10 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2024-03-21T08:40:09.466 INFO:teuthology.orchestra.run.smithi151.stdout: osd op complaint time = 180 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-21T08:40:09.467 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.468 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: mon op complaint time = 120 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-03-21T08:40:09.469 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: client mount timeout = 600 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: debug client = 20 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = "" 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: rados mon op timeout = 900 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout: rados osd op timeout = 900 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.a] 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.c] 2024-03-21T08:40:09.470 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.b] 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.d] 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout:[mds] 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: debug mds = 20 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: debug mds balancer = 20 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: mds debug frag = True 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: mds debug scatterstat = True 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: mds op complaint time = 180 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: mds verify scatter = True 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: osd op complaint time = 180 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: rados mon op timeout = 900 2024-03-21T08:40:09.471 INFO:teuthology.orchestra.run.smithi151.stdout: rados osd op timeout = 900 2024-03-21T08:40:09.473 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-21T08:40:09.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-21T08:40:09.574 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-21T08:40:09.575 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:09.575 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-21T08:40:09.583 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:09.583 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-21T08:40:09.632 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:40:09.632 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:09.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-21T08:40:09.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-21T08:40:09.703 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:09.704 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:09.752 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:40:09.752 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:09.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-21T08:40:09.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-21T08:40:09.787 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:09.787 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:09.837 INFO:tasks.ceph:Setting up mon nodes... 2024-03-21T08:40:09.838 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-21T08:40:09.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-21T08:40:09.893 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-21T08:40:09.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-21T08:40:09.956 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-21T08:40:09.970 INFO:tasks.ceph:Setting up mds nodes... 2024-03-21T08:40:09.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-21T08:40:10.026 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-21T08:40:10.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-21T08:40:10.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-21T08:40:10.148 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-21T08:40:10.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-21T08:40:10.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-21T08:40:10.228 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-21T08:40:10.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-21T08:40:10.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-21T08:40:10.346 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-21T08:40:10.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-21T08:40:10.370 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-21T08:40:10.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T08:40:10.413 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-21T08:40:10.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-21T08:40:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-21T08:40:10.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T08:40:10.577 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-21T08:40:10.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-21T08:40:10.682 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-21T08:40:10.698 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-21T08:40:10.699 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-21T08:40:10.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-21T08:40:10.718 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:10.718 INFO:tasks.ceph:role: osd.0 2024-03-21T08:40:10.719 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:40:10.719 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-21T08:40:10.779 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:10.780 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:10.804 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-03-21T08:40:10.807 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-03-21T08:40:10.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-21T08:40:10.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-21T08:40:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:10.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-21T08:40:10.999 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:11.000 INFO:tasks.ceph:role: osd.1 2024-03-21T08:40:11.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:40:11.000 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:11.059 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:11.060 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:11.060 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:11.084 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-03-21T08:40:11.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-03-21T08:40:11.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-21T08:40:11.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-21T08:40:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:11.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-21T08:40:11.284 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:11.284 INFO:tasks.ceph:role: osd.2 2024-03-21T08:40:11.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:40:11.284 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-21T08:40:11.343 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:11.343 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:11.343 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:11.343 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:11.344 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:11.367 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-03-21T08:40:11.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-03-21T08:40:11.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-21T08:40:11.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-21T08:40:11.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:11.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-21T08:40:11.563 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:11.564 INFO:tasks.ceph:role: osd.3 2024-03-21T08:40:11.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:40:11.564 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-21T08:40:11.623 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:11.623 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:11.623 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:11.624 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:11.648 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-03-21T08:40:11.650 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-03-21T08:40:11.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-21T08:40:11.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-21T08:40:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:11.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:11.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-21T08:40:11.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-21T08:40:11.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-21T08:40:11.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-21T08:40:11.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-21T08:40:11.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:11.898+0000 7fb6d29387c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-21T08:40:14.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T08:40:14.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.978+0000 7f16031da7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-21T08:40:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.978+0000 7f16031da7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-21T08:40:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.978+0000 7f16031da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-21T08:40:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.978+0000 7f16031da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-21T08:40:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.978+0000 7f16031da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-21T08:40:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:14.982+0000 7f16031da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-21T08:40:17.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T08:40:17.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:18.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-21T08:40:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-21T08:40:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-21T08:40:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-21T08:40:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-21T08:40:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:18.070+0000 7f50de2047c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-21T08:40:21.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T08:40:21.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-21T08:40:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-21T08:40:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-21T08:40:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-21T08:40:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-21T08:40:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:21.158+0000 7f856dc2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-21T08:40:24.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T08:40:24.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-21T08:40:24.189 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:24.189 INFO:tasks.ceph:role: osd.4 2024-03-21T08:40:24.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:40:24.189 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-21T08:40:24.255 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:24.256 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:24.742 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-21T08:40:24.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-21T08:40:24.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-21T08:40:24.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-21T08:40:24.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:24.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-21T08:40:24.989 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:24.989 INFO:tasks.ceph:role: osd.5 2024-03-21T08:40:24.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:40:24.990 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:25.065 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:25.492 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-21T08:40:25.495 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-21T08:40:25.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-21T08:40:25.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-21T08:40:25.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:25.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-21T08:40:25.681 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:25.681 INFO:tasks.ceph:role: osd.6 2024-03-21T08:40:25.682 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:40:25.682 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-21T08:40:25.753 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:25.753 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:25.754 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:26.216 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-21T08:40:26.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-21T08:40:26.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-21T08:40:26.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-21T08:40:26.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:26.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-21T08:40:26.422 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-21T08:40:26.422 INFO:tasks.ceph:role: osd.7 2024-03-21T08:40:26.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:40:26.422 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-21T08:40:26.493 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-21T08:40:27.000 INFO:teuthology.orchestra.run.smithi151.stdout:Discarding blocks...Done. 2024-03-21T08:40:27.003 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-03-21T08:40:27.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-21T08:40:27.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-21T08:40:27.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-03-21T08:40:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:40:27.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:27.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.317+0000 7efe48b5c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-21T08:40:27.325 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.321+0000 7efe48b5c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-21T08:40:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.321+0000 7efe48b5c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-21T08:40:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.321+0000 7efe48b5c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-21T08:40:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.321+0000 7efe48b5c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-21T08:40:27.327 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:27.325+0000 7efe48b5c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-21T08:40:30.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T08:40:30.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:30.431 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-21T08:40:30.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-21T08:40:30.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-21T08:40:30.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-21T08:40:30.432 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-21T08:40:30.433 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:30.429+0000 7f669d0b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-21T08:40:33.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T08:40:33.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.521+0000 7f8b174c17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-21T08:40:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.525+0000 7f8b174c17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-21T08:40:33.526 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.525+0000 7f8b174c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-21T08:40:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.525+0000 7f8b174c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-21T08:40:33.527 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.525+0000 7f8b174c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-21T08:40:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:33.525+0000 7f8b174c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-21T08:40:36.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T08:40:36.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:36.619 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-21T08:40:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-21T08:40:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-21T08:40:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-21T08:40:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-21T08:40:36.621 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:36.617+0000 7f18e66c77c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-21T08:40:37.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:40:37.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:40:39.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T08:40:39.615 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-21T08:40:39.615 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-21T08:40:39.638 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-21T08:40:39.701 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-21T08:40:39.765 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-21T08:40:39.829 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-21T08:40:39.893 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-21T08:40:39.957 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:39.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-21T08:40:40.021 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-21T08:40:40.043 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-21T08:40:40.105 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-21T08:40:40.165 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-21T08:40:40.225 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-21T08:40:40.289 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-21T08:40:40.349 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-21T08:40:40.409 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:40.409 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-21T08:40:40.418 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:40.418 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-21T08:40:40.469 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.469 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-21T08:40:40.478 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:40.478 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-21T08:40:40.530 INFO:tasks.ceph:Adding keys to all mons... 2024-03-21T08:40:40.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-21T08:40:40.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBp8vtl+il8NRAAUlToWvhBAZuAqKX8f58AOg== 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtl+trHARAAVeUAY/NXkRXJdfWXcNEaCg== 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlDRkWCRAAilhyLYX1c0vULfdMLzWWAg== 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBr8vtlK0S6NRAAKeHWfA4L/uoAhGbJeQ1p2w== 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-03-21T08:40:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBu8vtlGs2gOhAAFHS1xJ/R1i2EgFOJRLnTlw== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBy8vtl5TVhBBAAxqdX889ZTIy+UrQxJ5f2fg== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB18vtlpyuwCRAAUZq5F4NJnBLa2+w5o/JKyA== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBp8vtlA+USORAA7ZqV/pA9y1+4XBehbbumog== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlQySxDRAA2TpyxtlBbuQn0qvoIOJIQQ== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlnDLEFBAA6RwfJD0dSbvyuYmxKzSszA== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB78vtlOZsrExAAH0sYZ+QeHoo4NMVzEkGg3Q== 2024-03-21T08:40:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB+8vtld4LJGRAAl/n0wJSkXXTQBqPgxXyEHQ== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCB8vtlzRpuHxAAeRNXpyRn0AxBrPcyKN9WHw== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCE8vtlicn7JBAAv8yEBq2UMrr+rKEVspGVRQ== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[client.2] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlouglHxAAs+T0GLqKcfGvetFHJxp01Q== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtl4oOHIhAARXBoaT5pt4yfDvlIpcpweg== 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout:[client.3] 2024-03-21T08:40:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtljj+9KBAAA3rqu+R19oGWN/hi0PVTdQ== 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBp8vtl+il8NRAAUlToWvhBAZuAqKX8f58AOg== 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.a] 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtl+trHARAAVeUAY/NXkRXJdfWXcNEaCg== 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.c] 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtlDRkWCRAAilhyLYX1c0vULfdMLzWWAg== 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBr8vtlK0S6NRAAKeHWfA4L/uoAhGbJeQ1p2w== 2024-03-21T08:40:40.591 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBu8vtlGs2gOhAAFHS1xJ/R1i2EgFOJRLnTlw== 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBy8vtl5TVhBBAAxqdX889ZTIy+UrQxJ5f2fg== 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB18vtlpyuwCRAAUZq5F4NJnBLa2+w5o/JKyA== 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBp8vtlA+USORAA7ZqV/pA9y1+4XBehbbumog== 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.b] 2024-03-21T08:40:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtlQySxDRAA2TpyxtlBbuQn0qvoIOJIQQ== 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout:[mds.d] 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtlnDLEFBAA6RwfJD0dSbvyuYmxKzSszA== 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB78vtlOZsrExAAH0sYZ+QeHoo4NMVzEkGg3Q== 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQB+8vtld4LJGRAAl/n0wJSkXXTQBqPgxXyEHQ== 2024-03-21T08:40:40.593 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCB8vtlzRpuHxAAeRNXpyRn0AxBrPcyKN9WHw== 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCE8vtlicn7JBAAv8yEBq2UMrr+rKEVspGVRQ== 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:[client.2] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtlouglHxAAs+T0GLqKcfGvetFHJxp01Q== 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtl4oOHIhAARXBoaT5pt4yfDvlIpcpweg== 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout:[client.3] 2024-03-21T08:40:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBq8vtljj+9KBAAA3rqu+R19oGWN/hi0PVTdQ== 2024-03-21T08:40:40.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-21T08:40:40.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-21T08:40:40.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:40.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:40.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:40.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:40.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:40.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:40.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:40.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:40.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:40.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-21T08:40:41.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-21T08:40:41.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:41.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:41.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:41.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-21T08:40:41.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-21T08:40:41.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-21T08:40:41.732 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-21T08:40:41.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-21T08:40:41.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-21T08:40:41.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-21T08:40:41.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-21T08:40:41.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-21T08:40:42.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-21T08:40:42.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-21T08:40:42.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-21T08:40:42.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-21T08:40:42.227 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:42.231 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-21T08:40:42.280 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-21T08:40:42.281 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-21T08:40:42.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-21T08:40:42.284 INFO:tasks.ceph.mon.a:Started 2024-03-21T08:40:42.284 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-21T08:40:42.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-21T08:40:42.324 INFO:tasks.ceph.mon.b:Started 2024-03-21T08:40:42.324 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-21T08:40:42.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-21T08:40:42.327 INFO:tasks.ceph.mon.c:Started 2024-03-21T08:40:42.328 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-21T08:40:42.328 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-21T08:40:42.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-21T08:40:42.331 INFO:tasks.ceph.mgr.y:Started 2024-03-21T08:40:42.331 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-21T08:40:42.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-21T08:40:42.334 INFO:tasks.ceph.mgr.x:Started 2024-03-21T08:40:42.335 INFO:tasks.ceph:Setting crush tunables to default 2024-03-21T08:40:42.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-03-21T08:40:42.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.474+0000 7f4350200640 1 Processor -- start 2024-03-21T08:40:42.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.474+0000 7f4350200640 1 -- start start 2024-03-21T08:40:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f4350200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 0x7f434806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f4350200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f4350200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480b3010 con 0x7f434815e1e0 2024-03-21T08:40:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f4350200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434806d310 con 0x7f43480d0360 2024-03-21T08:40:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f4350200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434806d5f0 con 0x7f434806cbe0 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:39896/0 (socket says 172.21.15.28:39896) 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 learned_addr learned my addr 172.21.15.28:0/687122989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 msgr2=0x7f43480cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 msgr2=0x7f43480cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-21T08:40:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434ce00640 1 --2- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 0x7f434806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434ce00640 1 -- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 msgr2=0x7f434806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434ce00640 1 -- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 msgr2=0x7f434806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434ce00640 1 --2- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 0x7f434806c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-21T08:40:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.478+0000 7f434ce00640 1 --2- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 0x7f434806c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-21T08:40:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.630+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870080546 0 0) 0x7f43480b3010 con 0x7f434815e1e0 2024-03-21T08:40:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.630+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4330003530 con 0x7f434815e1e0 2024-03-21T08:40:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153939985 0 0) 0x7f4330003530 con 0x7f434815e1e0 2024-03-21T08:40:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 msgr2=0x7f43480cc4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43480cc4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 msgr2=0x7f434806c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 --2- 172.21.15.28:0/687122989 >> v2:172.21.15.151:3300/0 conn(0x7f43480d0360 0x7f434806c6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434806d8d0 con 0x7f434815e1e0 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f433c001680 con 0x7f434815e1e0 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 >> v1:172.21.15.151:6789/0 conn(0x7f434815e1e0 legacy=0x7f43480cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43300039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004040 0x7f4330004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4330004d50 0x7f43300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434806d310 con 0x7f4330004d50 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434806d5f0 con 0x7f4330004040 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433000bb30 con 0x7f434806cbe0 2024-03-21T08:40:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4330004d50 0x7f43300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434ce00640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004040 0x7f4330004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434d800640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43300039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 msgr2=0x7f43300039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f434806cbe0 0x7f43300039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004040 msgr2=0x7f4330004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004040 0x7f4330004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.634+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330003530 con 0x7f4330004d50 2024-03-21T08:40:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.638+0000 7f434e200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4330004d50 0x7f43300051d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4340009cc0 tx=0x7f434001cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4500f0d97214d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.638+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ab070 con 0x7f4330004d50 2024-03-21T08:40:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.638+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400a5aa0 con 0x7f4330004d50 2024-03-21T08:40:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.638+0000 7f4347e00640 1 -- 172.21.15.28:0/687122989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400098c0 con 0x7f4330004d50 2024-03-21T08:40:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- 172.21.15.28:0/687122989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4330004d50 msgr2=0x7f43300051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 --2- 172.21.15.28:0/687122989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4330004d50 0x7f43300051d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4340009cc0 tx=0x7f434001cb90 comp rx=0 tx=0).stop 2024-03-21T08:40:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434e200640 1 -- 172.21.15.28:0/687122989 reap_dead start 2024-03-21T08:40:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- 172.21.15.28:0/687122989 shutdown_connections 2024-03-21T08:40:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- 172.21.15.28:0/687122989 >> 172.21.15.28:0/687122989 conn(0x7f43480d1ab0 msgr2=0x7f43480d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- 172.21.15.28:0/687122989 shutdown_connections 2024-03-21T08:40:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- 172.21.15.28:0/687122989 wait complete. 2024-03-21T08:40:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 Processor -- start 2024-03-21T08:40:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- start start 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 0x7f434815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43480cad40 0x7f43480cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43480cc4e0 con 0x7f434815c920 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434806d010 con 0x7f4330004d50 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f4350200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43480cc660 con 0x7f43480cad40 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434d800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 0x7f434815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43480cad40 0x7f43480cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434d800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:34988/0 (socket says 172.21.15.28:34988) 2024-03-21T08:40:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.642+0000 7f434d800640 1 -- 172.21.15.28:0/4251477806 learned_addr learned my addr 172.21.15.28:0/4251477806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 -- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43480cad40 msgr2=0x7f43480cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43480cad40 0x7f43480cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 -- 172.21.15.28:0/4251477806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 msgr2=0x7f434815cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 0x7f434815cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 -- 172.21.15.28:0/4251477806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480cba70 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434ce00640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 0x7f434815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f434d800640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e7b0 tx=0x7f433c00ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c0050d0 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c018070 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43481f7e90 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c00c6c0 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43481f73b0 con 0x7f4330004d50 2024-03-21T08:40:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f433c00c860 con 0x7f4330004d50 2024-03-21T08:40:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f433c01d460 con 0x7f4330004d50 2024-03-21T08:40:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.646+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434815d2e0 con 0x7f4330004d50 2024-03-21T08:40:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.650+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f433c00cd30 con 0x7f4330004d50 2024-03-21T08:40:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.726+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f434804f4f0 con 0x7f4330004d50 2024-03-21T08:40:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.734+0000 7f4345600640 1 -- 172.21.15.28:0/4251477806 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f433c003600 con 0x7f4330004d50 2024-03-21T08:40:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-03-21T08:40:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 msgr2=0x7f43480ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f433c00e7b0 tx=0x7f433c00ebb0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 shutdown_connections 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f43480cad40 0x7f43480cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f434815c920 0x7f434815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 --2- 172.21.15.28:0/4251477806 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4330004d50 0x7f43480ca4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 >> 172.21.15.28:0/4251477806 conn(0x7f43480d1ab0 msgr2=0x7f434815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 shutdown_connections 2024-03-21T08:40:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:42.738+0000 7f4350200640 1 -- 172.21.15.28:0/4251477806 wait complete. 2024-03-21T08:40:42.755 INFO:tasks.ceph:check_enable_crimson: False 2024-03-21T08:40:42.756 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-21T08:40:42.756 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:42.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-21T08:40:42.778 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:42.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-21T08:40:42.839 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:42.837+0000 7f0c3969b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T08:40:42.844 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:42.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-21T08:40:42.849 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:42.850+0000 7f1b5f09b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T08:40:42.868 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:40:42.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-21T08:40:42.927 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:42.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-21T08:40:42.947 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:42.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-21T08:40:43.013 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:43.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-21T08:40:43.076 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:40:43.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-21T08:40:43.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 7b33d81c-4e5b-47ee-8d63-94fc0d309fdc 0 2024-03-21T08:40:43.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 Processor -- start 2024-03-21T08:40:43.312 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 -- start start 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff98415bd20 0x7ff9840b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98415fba0 con 0x7ff9840b0e80 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98415fe80 con 0x7ff98415bd20 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.309+0000 7ff98ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff984160160 con 0x7ff9840cebd0 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:53260/0 (socket says 172.21.15.151:53260) 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 -- 172.21.15.151:0/1754077491 learned_addr learned my addr 172.21.15.151:0/1754077491 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.314 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff983e00640 1 --2- 172.21.15.151:0/1754077491 >> v2:172.21.15.28:3300/0 conn(0x7ff98415bd20 0x7ff9840b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 -- 172.21.15.151:0/1754077491 >> v2:172.21.15.28:3300/0 conn(0x7ff98415bd20 msgr2=0x7ff9840b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:43.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 --2- 172.21.15.151:0/1754077491 >> v2:172.21.15.28:3300/0 conn(0x7ff98415bd20 0x7ff9840b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 -- 172.21.15.151:0/1754077491 >> v1:172.21.15.28:6789/0 conn(0x7ff9840b0e80 legacy=0x7ff9840ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.315 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 -- 172.21.15.151:0/1754077491 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984160440 con 0x7ff9840cebd0 2024-03-21T08:40:43.317 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff988c00640 1 --2- 172.21.15.151:0/1754077491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009e50 tx=0x7ff97c01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b77f89964e48926b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff982a00640 1 -- 172.21.15.151:0/1754077491 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0a5700 con 0x7ff9840cebd0 2024-03-21T08:40:43.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff982a00640 1 -- 172.21.15.151:0/1754077491 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c0a58a0 con 0x7ff9840cebd0 2024-03-21T08:40:43.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.313+0000 7ff982a00640 1 -- 172.21.15.151:0/1754077491 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0a5c70 con 0x7ff9840cebd0 2024-03-21T08:40:43.318 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- 172.21.15.151:0/1754077491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 msgr2=0x7ff98408afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- 172.21.15.151:0/1754077491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009e50 tx=0x7ff97c01ca30 comp rx=0 tx=0).stop 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- 172.21.15.151:0/1754077491 shutdown_connections 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- 172.21.15.151:0/1754077491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840cebd0 0x7ff98408afc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- 172.21.15.151:0/1754077491 >> v2:172.21.15.28:3300/0 conn(0x7ff98415bd20 0x7ff9840b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- 172.21.15.151:0/1754077491 >> 172.21.15.151:0/1754077491 conn(0x7ff9840d1ab0 msgr2=0x7ff9840d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- 172.21.15.151:0/1754077491 shutdown_connections 2024-03-21T08:40:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- 172.21.15.151:0/1754077491 wait complete. 2024-03-21T08:40:43.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 Processor -- start 2024-03-21T08:40:43.320 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- start start 2024-03-21T08:40:43.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 0x7ff9840ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.321 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98415bd20 0x7ff9840afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98408be90 con 0x7ff98415bd20 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98408bd10 con 0x7ff9840cebd0 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff98ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff984168d10 con 0x7ff9840b0e80 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff983400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff983e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 0x7ff9840ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff983400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:49122/0 (socket says 172.21.15.151:49122) 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff983400640 1 -- 172.21.15.151:0/495853664 learned_addr learned my addr 172.21.15.151:0/495853664 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.317+0000 7ff988c00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98415bd20 0x7ff9840afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.323 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 -- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 msgr2=0x7ff984154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 -- 172.21.15.151:0/495853664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98415bd20 msgr2=0x7ff9840afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98415bd20 0x7ff9840afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 -- 172.21.15.151:0/495853664 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840c9590 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983400640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff983e00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 0x7ff9840ca910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e8a0 tx=0x7ff97400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974017070 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97400c940 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97400ccf0 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9840c9870 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9840c9e30 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff97401a070 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff97401c7d0 con 0x7ff9840b0e80 2024-03-21T08:40:43.324 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.321+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98404f4f0 con 0x7ff9840b0e80 2024-03-21T08:40:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.325+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff974013470 con 0x7ff9840b0e80 2024-03-21T08:40:43.356 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:43.353+0000 7f0c3969b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T08:40:43.376 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:43.374+0000 7f1b5f09b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T08:40:43.393 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.389+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7b33d81c-4e5b-47ee-8d63-94fc0d309fdc", "id": 0} v 0) v1 -- 0x7ff9840d2830 con 0x7ff9840b0e80 2024-03-21T08:40:43.399 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff97be00640 1 -- 172.21.15.151:0/495853664 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7b33d81c-4e5b-47ee-8d63-94fc0d309fdc", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff97401cb00 con 0x7ff9840b0e80 2024-03-21T08:40:43.399 INFO:teuthology.orchestra.run.smithi151.stdout:0 2024-03-21T08:40:43.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 msgr2=0x7ff9840ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 0x7ff9840ca910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e8a0 tx=0x7ff97400ed60 comp rx=0 tx=0).stop 2024-03-21T08:40:43.400 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 shutdown_connections 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98415bd20 0x7ff9840afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9840cebd0 0x7ff984154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 --2- 172.21.15.151:0/495853664 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff9840b0e80 0x7ff9840ca910 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 >> 172.21.15.151:0/495853664 conn(0x7ff9840d1ab0 msgr2=0x7ff984073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 shutdown_connections 2024-03-21T08:40:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.397+0000 7ff98ac00640 1 -- 172.21.15.151:0/495853664 wait complete. 2024-03-21T08:40:43.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new e302143c-aff4-4d4a-8816-61a78f96bd2a 1 2024-03-21T08:40:43.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.549+0000 7f09d3200640 1 Processor -- start 2024-03-21T08:40:43.553 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.549+0000 7f09d3200640 1 -- start start 2024-03-21T08:40:43.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09d3200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f09cc15bd20 0x7f09cc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.555 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09d3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09cc0b0e80 0x7f09cc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09d3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0cede0 con 0x7f09cc0cea00 2024-03-21T08:40:43.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09d3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc08bfd0 con 0x7f09cc0b0e80 2024-03-21T08:40:43.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09d3200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc15fba0 con 0x7f09cc15bd20 2024-03-21T08:40:43.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f09cc15bd20 0x7f09cc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.556 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f09cc15bd20 0x7f09cc0b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55860/0 (socket says 172.21.15.151:55860) 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 -- 172.21.15.151:0/3523705898 learned_addr learned my addr 172.21.15.151:0/3523705898 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09c3e00640 1 --2- 172.21.15.151:0/3523705898 >> v2:172.21.15.28:3300/0 conn(0x7f09cc0b0e80 0x7f09cc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 -- 172.21.15.151:0/3523705898 >> v2:172.21.15.28:3300/0 conn(0x7f09cc0b0e80 msgr2=0x7f09cc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 --2- 172.21.15.151:0/3523705898 >> v2:172.21.15.28:3300/0 conn(0x7f09cc0b0e80 0x7f09cc0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 -- 172.21.15.151:0/3523705898 >> v1:172.21.15.28:6789/0 conn(0x7f09cc0cea00 legacy=0x7f09cc0b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.553+0000 7f09cbe00640 1 -- 172.21.15.151:0/3523705898 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc15fe80 con 0x7f09cc15bd20 2024-03-21T08:40:43.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09cbe00640 1 --2- 172.21.15.151:0/3523705898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc15bd20 0x7f09cc0b2c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f09cc068b20 tx=0x7f09bc01c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6033c1644aedbdd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09cb400640 1 -- 172.21.15.151:0/3523705898 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0b0070 con 0x7f09cc15bd20 2024-03-21T08:40:43.558 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09cb400640 1 -- 172.21.15.151:0/3523705898 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc0ab600 con 0x7f09cc15bd20 2024-03-21T08:40:43.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09cb400640 1 -- 172.21.15.151:0/3523705898 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0ab9d0 con 0x7f09cc15bd20 2024-03-21T08:40:43.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- 172.21.15.151:0/3523705898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc15bd20 msgr2=0x7f09cc0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.559 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- 172.21.15.151:0/3523705898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc15bd20 0x7f09cc0b2c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f09cc068b20 tx=0x7f09bc01c520 comp rx=0 tx=0).stop 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- 172.21.15.151:0/3523705898 shutdown_connections 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- 172.21.15.151:0/3523705898 >> v2:172.21.15.28:3300/0 conn(0x7f09cc0b0e80 0x7f09cc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- 172.21.15.151:0/3523705898 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc15bd20 0x7f09cc0b2c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- 172.21.15.151:0/3523705898 >> 172.21.15.151:0/3523705898 conn(0x7f09cc0d1ab0 msgr2=0x7f09cc0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- 172.21.15.151:0/3523705898 shutdown_connections 2024-03-21T08:40:43.560 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- 172.21.15.151:0/3523705898 wait complete. 2024-03-21T08:40:43.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 Processor -- start 2024-03-21T08:40:43.561 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- start start 2024-03-21T08:40:43.562 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0b0e80 0x7f09cc070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09cc15bd20 0x7f09cc0c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc0cc6a0 con 0x7f09cc0b0e80 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc0cc520 con 0x7f09cc0cea00 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.557+0000 7f09d3200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc15c100 con 0x7f09cc15bd20 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09d1200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09cc15bd20 0x7f09cc0c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55872/0 (socket says 172.21.15.151:55872) 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 -- 172.21.15.151:0/3153126709 learned_addr learned my addr 172.21.15.151:0/3153126709 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09cbe00640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0b0e80 0x7f09cc070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 -- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09cc15bd20 msgr2=0x7f09cc0c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09cc15bd20 0x7f09cc0c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.563 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 -- 172.21.15.151:0/3153126709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0b0e80 msgr2=0x7f09cc070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0b0e80 0x7f09cc070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 -- 172.21.15.151:0/3153126709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0ca330 con 0x7f09cc0cea00 2024-03-21T08:40:43.564 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c3e00640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09b400e9b0 tx=0x7f09b400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4017070 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b400c9a0 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc0ca610 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b400cd70 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc0cabd0 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f09b401a070 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f09b401c9c0 con 0x7f09cc0cea00 2024-03-21T08:40:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.561+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc04f4f0 con 0x7f09cc0cea00 2024-03-21T08:40:43.566 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.565+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f09b401cbc0 con 0x7f09cc0cea00 2024-03-21T08:40:43.583 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:43.581+0000 7f0c3969b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T08:40:43.602 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:43.602+0000 7f1b5f09b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T08:40:43.634 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.633+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e302143c-aff4-4d4a-8816-61a78f96bd2a", "id": 1} v 0) v1 -- 0x7f09cc0d2830 con 0x7f09cc0cea00 2024-03-21T08:40:43.640 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09c8c00640 1 -- 172.21.15.151:0/3153126709 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e302143c-aff4-4d4a-8816-61a78f96bd2a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09b40059a0 con 0x7f09cc0cea00 2024-03-21T08:40:43.641 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 msgr2=0x7f09cc151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09b400e9b0 tx=0x7f09b400ee70 comp rx=0 tx=0).stop 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 shutdown_connections 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f09cc15bd20 0x7f09cc0c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f09cc0cea00 0x7f09cc151f20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 --2- 172.21.15.151:0/3153126709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0b0e80 0x7f09cc070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 >> 172.21.15.151:0/3153126709 conn(0x7f09cc0d1ab0 msgr2=0x7f09cc0b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.642 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 shutdown_connections 2024-03-21T08:40:43.643 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.637+0000 7f09d3200640 1 -- 172.21.15.151:0/3153126709 wait complete. 2024-03-21T08:40:43.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc 2 2024-03-21T08:40:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.789+0000 7f6f0b800640 1 Processor -- start 2024-03-21T08:40:43.794 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.789+0000 7f6f0b800640 1 -- start start 2024-03-21T08:40:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f0b800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6f0415f890 0x7f6f0415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.795 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f0b800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 0x7f6f040ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f0b800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f040cee70 con 0x7f6f0415bc90 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f0b800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f040cfe80 con 0x7f6f0415f890 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f0b800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04160400 con 0x7f6f040ce530 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6f0415f890 0x7f6f0415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f09800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 0x7f6f040ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6f0415f890 0x7f6f0415fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55882/0 (socket says 172.21.15.151:55882) 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f08e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6f0415bc90 0x7f6f0408bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42228/0 (socket says 172.21.15.151:42228) 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 -- 172.21.15.151:0/1546001461 learned_addr learned my addr 172.21.15.151:0/1546001461 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.796 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 -- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 msgr2=0x7f6f040ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 0x7f6f040ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 -- 172.21.15.151:0/1546001461 >> v1:172.21.15.151:6789/0 conn(0x7f6f0415bc90 legacy=0x7f6f0408bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03e00640 1 -- 172.21.15.151:0/1546001461 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0408b060 con 0x7f6f0415f890 2024-03-21T08:40:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f03400640 1 -- 172.21.15.151:0/1546001461 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476154285 0 0) 0x7f6f040cee70 con 0x7f6f0415bc90 2024-03-21T08:40:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.793+0000 7f6f09800640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 0x7f6f040ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:43.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f03e00640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f0415f890 0x7f6f0415fce0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009870 tx=0x7f6ef001ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5ff8d924180bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f03400640 1 -- 172.21.15.151:0/1546001461 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00b5070 con 0x7f6f0415f890 2024-03-21T08:40:43.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f03400640 1 -- 172.21.15.151:0/1546001461 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00a8a50 con 0x7f6f0415f890 2024-03-21T08:40:43.799 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f03400640 1 -- 172.21.15.151:0/1546001461 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00a8e20 con 0x7f6f0415f890 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f0415f890 msgr2=0x7f6f0415fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f0415f890 0x7f6f0415fce0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009870 tx=0x7f6ef001ed50 comp rx=0 tx=0).stop 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- 172.21.15.151:0/1546001461 shutdown_connections 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f040ce530 0x7f6f040ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1546001461 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f0415f890 0x7f6f0415fce0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- 172.21.15.151:0/1546001461 >> 172.21.15.151:0/1546001461 conn(0x7f6f040d1b20 msgr2=0x7f6f040d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- 172.21.15.151:0/1546001461 shutdown_connections 2024-03-21T08:40:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- 172.21.15.151:0/1546001461 wait complete. 2024-03-21T08:40:43.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 Processor -- start 2024-03-21T08:40:43.801 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.797+0000 7f6f0b800640 1 -- start start 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f0415f890 0x7f6f0415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041606e0 con 0x7f6f0415f890 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0408b620 con 0x7f6f040ce530 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0415c070 con 0x7f6f0415bc90 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.804 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:45464/0 (socket says 172.21.15.151:45464) 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f03e00640 1 -- 172.21.15.151:0/1115113457 learned_addr learned my addr 172.21.15.151:0/1115113457 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55890/0 (socket says 172.21.15.151:55890) 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f09800640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f0415f890 0x7f6f0415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 -- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 msgr2=0x7f6f0415a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 -- 172.21.15.151:0/1115113457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f0415f890 msgr2=0x7f6f0415e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f0415f890 0x7f6f0415e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 -- 172.21.15.151:0/1115113457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f04159730 con 0x7f6f040ce530 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f03e00640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f08e00640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c430 tx=0x7f6ef800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80074e0 con 0x7f6f040ce530 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef80047f0 con 0x7f6f040ce530 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8004be0 con 0x7f6f040ce530 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f04159a10 con 0x7f6f040ce530 2024-03-21T08:40:43.805 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f04159fd0 con 0x7f6f040ce530 2024-03-21T08:40:43.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6ef8011070 con 0x7f6f040ce530 2024-03-21T08:40:43.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f6ef8004d80 con 0x7f6f040ce530 2024-03-21T08:40:43.806 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.801+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0404f4f0 con 0x7f6f040ce530 2024-03-21T08:40:43.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.805+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6ef801d3f0 con 0x7f6f040ce530 2024-03-21T08:40:43.873 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.869+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc", "id": 2} v 0) v1 -- 0x7f6f040b1730 con 0x7f6f040ce530 2024-03-21T08:40:43.877 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.873+0000 7f6f00c00640 1 -- 172.21.15.151:0/1115113457 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6ef801d760 con 0x7f6f040ce530 2024-03-21T08:40:43.877 INFO:teuthology.orchestra.run.smithi151.stdout:2 2024-03-21T08:40:43.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 msgr2=0x7f6f041590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:43.878 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c430 tx=0x7f6ef800c830 comp rx=0 tx=0).stop 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 shutdown_connections 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f0415f890 0x7f6f0415e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f0415bc90 0x7f6f0415a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 --2- 172.21.15.151:0/1115113457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f040ce530 0x7f6f041590b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 >> 172.21.15.151:0/1115113457 conn(0x7f6f040d1b20 msgr2=0x7f6f04070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 shutdown_connections 2024-03-21T08:40:43.879 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:43.877+0000 7f6f0b800640 1 -- 172.21.15.151:0/1115113457 wait complete. 2024-03-21T08:40:43.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new cc23f2f0-5e4e-48a8-a39b-84c0451e4630 3 2024-03-21T08:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 Processor -- start 2024-03-21T08:40:44.031 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 -- start start 2024-03-21T08:40:44.032 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a00066ef0 con 0x7f1a0015bb70 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a000671d0 con 0x7f1a000d0640 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a08400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000b5490 con 0x7f1a000b5030 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:45470/0 (socket says 172.21.15.151:45470) 2024-03-21T08:40:44.033 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 -- 172.21.15.151:0/3146968300 learned_addr learned my addr 172.21.15.151:0/3146968300 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:44.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 -- 172.21.15.151:0/3146968300 >> v1:172.21.15.151:6789/0 conn(0x7f1a000d0640 legacy=0x7f1a000b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 -- 172.21.15.151:0/3146968300 >> v1:172.21.15.28:6789/0 conn(0x7f1a0015bb70 legacy=0x7f1a000cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.029+0000 7f1a05000640 1 -- 172.21.15.151:0/3146968300 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a000674b0 con 0x7f1a000b5030 2024-03-21T08:40:44.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a05000640 1 --2- 172.21.15.151:0/3146968300 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0098a0 tx=0x7f19ec01c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3845fb8b13f39433 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f19ffe00640 1 -- 172.21.15.151:0/3146968300 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0b0070 con 0x7f1a000b5030 2024-03-21T08:40:44.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f19ffe00640 1 -- 172.21.15.151:0/3146968300 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec0ab600 con 0x7f1a000b5030 2024-03-21T08:40:44.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f19ffe00640 1 -- 172.21.15.151:0/3146968300 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0ab9d0 con 0x7f1a000b5030 2024-03-21T08:40:44.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 -- 172.21.15.151:0/3146968300 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 msgr2=0x7f1a0006c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 --2- 172.21.15.151:0/3146968300 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0098a0 tx=0x7f19ec01c4f0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 -- 172.21.15.151:0/3146968300 shutdown_connections 2024-03-21T08:40:44.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 --2- 172.21.15.151:0/3146968300 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a000b5030 0x7f1a0006c390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.037 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 -- 172.21.15.151:0/3146968300 >> 172.21.15.151:0/3146968300 conn(0x7f1a00099a30 msgr2=0x7f1a000b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 -- 172.21.15.151:0/3146968300 shutdown_connections 2024-03-21T08:40:44.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.033+0000 7f1a08400640 1 -- 172.21.15.151:0/3146968300 wait complete. 2024-03-21T08:40:44.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 Processor -- start 2024-03-21T08:40:44.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- start start 2024-03-21T08:40:44.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a000b5030 0x7f1a0014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a0015bb70 0x7f1a00152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0015bf50 con 0x7f1a000b5030 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000ccff0 con 0x7f1a000d0640 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0015c0d0 con 0x7f1a0015bb70 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55900/0 (socket says 172.21.15.151:55900) 2024-03-21T08:40:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 -- 172.21.15.151:0/2034520515 learned_addr learned my addr 172.21.15.151:0/2034520515 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a06400640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a0015bb70 0x7f1a00152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05a00640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a000b5030 0x7f1a0014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 -- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a0015bb70 msgr2=0x7f1a00152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a0015bb70 0x7f1a00152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 -- 172.21.15.151:0/2034520515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a000b5030 msgr2=0x7f1a0014acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a000b5030 0x7f1a0014acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 -- 172.21.15.151:0/2034520515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00158ad0 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a05000640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0ae040 tx=0x7f19ec019290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0b8020 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec0b4660 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0b0040 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a000ad250 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a00158db0 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f19ec0bb070 con 0x7f1a000d0640 2024-03-21T08:40:44.043 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.037+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f19ec0ab4c0 con 0x7f1a000d0640 2024-03-21T08:40:44.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.041+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0009e530 con 0x7f1a000d0640 2024-03-21T08:40:44.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.041+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f19ec016170 con 0x7f1a000d0640 2024-03-21T08:40:44.111 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.109+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cc23f2f0-5e4e-48a8-a39b-84c0451e4630", "id": 3} v 0) v1 -- 0x7f1a0004f4f0 con 0x7f1a000d0640 2024-03-21T08:40:44.115 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f19fd600640 1 -- 172.21.15.151:0/2034520515 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cc23f2f0-5e4e-48a8-a39b-84c0451e4630", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f19ec0b4a50 con 0x7f1a000d0640 2024-03-21T08:40:44.115 INFO:teuthology.orchestra.run.smithi151.stdout:3 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 msgr2=0x7f1a00158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f19ec0ae040 tx=0x7f19ec019290 comp rx=0 tx=0).stop 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 shutdown_connections 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a0015bb70 0x7f1a00152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a000d0640 0x7f1a00158060 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 --2- 172.21.15.151:0/2034520515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a000b5030 0x7f1a0014acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 >> 172.21.15.151:0/2034520515 conn(0x7f1a00099a30 msgr2=0x7f1a000b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 shutdown_connections 2024-03-21T08:40:44.117 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.113+0000 7f1a08400640 1 -- 172.21.15.151:0/2034520515 wait complete. 2024-03-21T08:40:44.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 8f556bd4-249e-46ed-9ce0-6ac5e31994e2 4 2024-03-21T08:40:44.218 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.217+0000 7f0c3969b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T08:40:44.243 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.241+0000 7f1b5f09b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T08:40:44.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 Processor -- start 2024-03-21T08:40:44.272 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 -- start start 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69b80cd4f0 0x7f69b8168f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b80b3010 con 0x7f69b80d0360 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b806d310 con 0x7f69b815e1e0 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bfc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b806d5f0 con 0x7f69b80cd4f0 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bd200640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f69b815e1e0 0x7f69b80cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42252/0 (socket says 172.21.15.151:42252) 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bd200640 1 -- 172.21.15.151:0/1343269860 learned_addr learned my addr 172.21.15.151:0/1343269860 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-21T08:40:44.273 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69bdc00640 1 --2- 172.21.15.151:0/1343269860 >> v2:172.21.15.28:3300/0 conn(0x7f69b80cd4f0 0x7f69b8168f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.269+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870489934 0 0) 0x7f69b806d310 con 0x7f69b815e1e0 2024-03-21T08:40:44.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a0003530 con 0x7f69b815e1e0 2024-03-21T08:40:44.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bdc00640 1 -- 172.21.15.151:0/1343269860 >> v1:172.21.15.151:6789/0 conn(0x7f69b815e1e0 legacy=0x7f69b80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.274 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072134879 0 0) 0x7f69b80b3010 con 0x7f69b80d0360 2024-03-21T08:40:44.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bdc00640 1 -- 172.21.15.151:0/1343269860 >> v1:172.21.15.28:6789/0 conn(0x7f69b80d0360 legacy=0x7f69b806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.275 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bdc00640 1 -- 172.21.15.151:0/1343269860 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b806d8d0 con 0x7f69b80cd4f0 2024-03-21T08:40:44.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bdc00640 1 --2- 172.21.15.151:0/1343269860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80cd4f0 0x7f69b8168f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3678cf13910c577 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00b2070 con 0x7f69b80cd4f0 2024-03-21T08:40:44.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b00a5940 con 0x7f69b80cd4f0 2024-03-21T08:40:44.277 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69b7400640 1 -- 172.21.15.151:0/1343269860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00a5d10 con 0x7f69b80cd4f0 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bfc00640 1 -- 172.21.15.151:0/1343269860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80cd4f0 msgr2=0x7f69b8168f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bfc00640 1 --2- 172.21.15.151:0/1343269860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80cd4f0 0x7f69b8168f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b001c970 comp rx=0 tx=0).stop 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bfc00640 1 -- 172.21.15.151:0/1343269860 shutdown_connections 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.273+0000 7f69bfc00640 1 --2- 172.21.15.151:0/1343269860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80cd4f0 0x7f69b8168f30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- 172.21.15.151:0/1343269860 >> 172.21.15.151:0/1343269860 conn(0x7f69b80d1ab0 msgr2=0x7f69b80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- 172.21.15.151:0/1343269860 shutdown_connections 2024-03-21T08:40:44.278 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- 172.21.15.151:0/1343269860 wait complete. 2024-03-21T08:40:44.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 Processor -- start 2024-03-21T08:40:44.279 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- start start 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80d0360 0x7f69b814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 0x7f69b80c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b816c020 con 0x7f69b80d0360 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b816bea0 con 0x7f69b80cd4f0 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bfc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b816c1a0 con 0x7f69b815e1e0 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bd200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.281 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 0x7f69b80c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bd200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55908/0 (socket says 172.21.15.151:55908) 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bd200640 1 -- 172.21.15.151:0/2203527267 learned_addr learned my addr 172.21.15.151:0/2203527267 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69b7e00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80d0360 0x7f69b814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 -- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 msgr2=0x7f69b80b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 -- 172.21.15.151:0/2203527267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80d0360 msgr2=0x7f69b814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80d0360 0x7f69b814ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 -- 172.21.15.151:0/2203527267 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b814d3a0 con 0x7f69b815e1e0 2024-03-21T08:40:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bd200640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:44.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.277+0000 7f69bdc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 0x7f69b80c8b80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69b00ac000 tx=0x7f69b0019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00bd020 con 0x7f69b815e1e0 2024-03-21T08:40:44.283 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b00b3040 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00b2050 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b814d7a0 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b80b6f60 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f69b00c0070 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f69b00b9450 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b804f4f0 con 0x7f69b815e1e0 2024-03-21T08:40:44.284 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f0c3969b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T08:40:44.285 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.281+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f69b0016560 con 0x7f69b815e1e0 2024-03-21T08:40:44.309 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.309+0000 7f1b5f09b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T08:40:44.347 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.345+0000 7f0c3969b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T08:40:44.351 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.349+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8f556bd4-249e-46ed-9ce0-6ac5e31994e2", "id": 4} v 0) v1 -- 0x7f69b80d2830 con 0x7f69b815e1e0 2024-03-21T08:40:44.356 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69b4c00640 1 -- 172.21.15.151:0/2203527267 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f556bd4-249e-46ed-9ce0-6ac5e31994e2", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f69b00adc00 con 0x7f69b815e1e0 2024-03-21T08:40:44.356 INFO:teuthology.orchestra.run.smithi151.stdout:4 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 msgr2=0x7f69b80c8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 0x7f69b80c8b80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69b00ac000 tx=0x7f69b0019710 comp rx=0 tx=0).stop 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 shutdown_connections 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f69b815e1e0 0x7f69b80c8b80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b80d0360 0x7f69b814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 --2- 172.21.15.151:0/2203527267 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f69b80cd4f0 0x7f69b80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 >> 172.21.15.151:0/2203527267 conn(0x7f69b80d1ab0 msgr2=0x7f69b815a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.358 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.353+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 shutdown_connections 2024-03-21T08:40:44.359 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.357+0000 7f69bfc00640 1 -- 172.21.15.151:0/2203527267 wait complete. 2024-03-21T08:40:44.374 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.373+0000 7f1b5f09b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T08:40:44.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 6d587fed-3a5a-402c-b87e-d16a3c1d36aa 5 2024-03-21T08:40:44.441 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.437+0000 7f0c3969b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T08:40:44.468 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.465+0000 7f1b5f09b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T08:40:44.502 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.501+0000 7f0c3969b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T08:40:44.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 Processor -- start 2024-03-21T08:40:44.511 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 -- start start 2024-03-21T08:40:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a080af910 0x7f8a080afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a080d1790 con 0x7f8a0806eb40 2024-03-21T08:40:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a080cd7d0 con 0x7f8a080d13b0 2024-03-21T08:40:44.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a10800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a080cdab0 con 0x7f8a080af910 2024-03-21T08:40:44.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0de00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f8a080d13b0 0x7f8a08161ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42268/0 (socket says 172.21.15.151:42268) 2024-03-21T08:40:44.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0de00640 1 -- 172.21.15.151:0/2350370495 learned_addr learned my addr 172.21.15.151:0/2350370495 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-21T08:40:44.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0d400640 1 --2- 172.21.15.151:0/2350370495 >> v2:172.21.15.28:3300/0 conn(0x7f8a080af910 0x7f8a080afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.513 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204508447 0 0) 0x7f8a080cd7d0 con 0x7f8a080d13b0 2024-03-21T08:40:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f0003530 con 0x7f8a080d13b0 2024-03-21T08:40:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264522023 0 0) 0x7f8a080d1790 con 0x7f8a0806eb40 2024-03-21T08:40:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0d400640 1 -- 172.21.15.151:0/2350370495 >> v1:172.21.15.151:6789/0 conn(0x7f8a080d13b0 legacy=0x7f8a08161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0d400640 1 -- 172.21.15.151:0/2350370495 >> v1:172.21.15.28:6789/0 conn(0x7f8a0806eb40 legacy=0x7f8a0806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.514 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.509+0000 7f8a0d400640 1 -- 172.21.15.151:0/2350370495 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a080cdd90 con 0x7f8a080af910 2024-03-21T08:40:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a0d400640 1 --2- 172.21.15.151:0/2350370495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a080afda0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097f0 tx=0x7f89f801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2853e6ac174b5e62 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f80b1070 con 0x7f8a080af910 2024-03-21T08:40:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f80ac410 con 0x7f8a080af910 2024-03-21T08:40:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a0ca00640 1 -- 172.21.15.151:0/2350370495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f80ac7e0 con 0x7f8a080af910 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 -- 172.21.15.151:0/2350370495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 msgr2=0x7f8a080afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 --2- 172.21.15.151:0/2350370495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a080afda0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097f0 tx=0x7f89f801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 -- 172.21.15.151:0/2350370495 shutdown_connections 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 --2- 172.21.15.151:0/2350370495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a080afda0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 -- 172.21.15.151:0/2350370495 >> 172.21.15.151:0/2350370495 conn(0x7f8a08099a30 msgr2=0x7f8a08099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 -- 172.21.15.151:0/2350370495 shutdown_connections 2024-03-21T08:40:44.517 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.513+0000 7f8a10800640 1 -- 172.21.15.151:0/2350370495 wait complete. 2024-03-21T08:40:44.518 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 Processor -- start 2024-03-21T08:40:44.519 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- start start 2024-03-21T08:40:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a0814e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a080d13b0 0x7f8a0814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0806ba00 con 0x7f8a080af910 2024-03-21T08:40:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0806b880 con 0x7f8a0806eb40 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0815edf0 con 0x7f8a080d13b0 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55924/0 (socket says 172.21.15.151:55924) 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a080d13b0 0x7f8a0814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 -- 172.21.15.151:0/1324694803 learned_addr learned my addr 172.21.15.151:0/1324694803 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0d400640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a0814e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 -- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a080d13b0 msgr2=0x7f8a0814e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a080d13b0 0x7f8a0814e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 -- 172.21.15.151:0/1324694803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 msgr2=0x7f8a0814e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a0814e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 -- 172.21.15.151:0/1324694803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08156b50 con 0x7f8a0806eb40 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0d400640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a0814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:44.522 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a0de00640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c3c0 tx=0x7f89fc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc017070 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc005cf0 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0072f0 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a08156df0 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a081573b0 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f89fc01a070 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.517+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f89fc007490 con 0x7f8a0806eb40 2024-03-21T08:40:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.521+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0809ea60 con 0x7f8a0806eb40 2024-03-21T08:40:44.524 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.521+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f89fc021020 con 0x7f8a0806eb40 2024-03-21T08:40:44.531 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.529+0000 7f1b5f09b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T08:40:44.590 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.589+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6d587fed-3a5a-402c-b87e-d16a3c1d36aa", "id": 5} v 0) v1 -- 0x7f8a0816a9e0 con 0x7f8a0806eb40 2024-03-21T08:40:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a02000640 1 -- 172.21.15.151:0/1324694803 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d587fed-3a5a-402c-b87e-d16a3c1d36aa", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f89fc01c440 con 0x7f8a0806eb40 2024-03-21T08:40:44.595 INFO:teuthology.orchestra.run.smithi151.stdout:5 2024-03-21T08:40:44.595 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f0c3969b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T08:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 msgr2=0x7f8a08152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c3c0 tx=0x7f89fc00c880 comp rx=0 tx=0).stop 2024-03-21T08:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 shutdown_connections 2024-03-21T08:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8a080d13b0 0x7f8a0814e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.596 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a080af910 0x7f8a0814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 --2- 172.21.15.151:0/1324694803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8a0806eb40 0x7f8a08152b90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 >> 172.21.15.151:0/1324694803 conn(0x7f8a08099a30 msgr2=0x7f8a08159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 shutdown_connections 2024-03-21T08:40:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.593+0000 7f8a10800640 1 -- 172.21.15.151:0/1324694803 wait complete. 2024-03-21T08:40:44.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new a89375db-4a99-4718-b176-616941009b89 6 2024-03-21T08:40:44.626 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.625+0000 7f1b5f09b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T08:40:44.663 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.661+0000 7f0c3969b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T08:40:44.693 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.693+0000 7f1b5f09b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T08:40:44.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 Processor -- start 2024-03-21T08:40:44.751 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 -- start start 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9880cd4f0 0x7ff988168f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880b3010 con 0x7ff9880d0360 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98806d310 con 0x7ff98815e1e0 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff98da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98806d5f0 con 0x7ff9880cd4f0 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff986a00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff98815e1e0 0x7ff9880cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.151:42282/0 (socket says 172.21.15.151:42282) 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff986a00640 1 -- 172.21.15.151:0/2331771195 learned_addr learned my addr 172.21.15.151:0/2331771195 (peer_addr_for_me v1:172.21.15.151:0/0) 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff987400640 1 --2- 172.21.15.151:0/2331771195 >> v2:172.21.15.28:3300/0 conn(0x7ff9880cd4f0 0x7ff988168f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324957494 0 0) 0x7ff98806d310 con 0x7ff98815e1e0 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c003530 con 0x7ff98815e1e0 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177806600 0 0) 0x7ff9880b3010 con 0x7ff9880d0360 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff98806d310 con 0x7ff9880d0360 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff987400640 1 -- 172.21.15.151:0/2331771195 >> v1:172.21.15.151:6789/0 conn(0x7ff98815e1e0 legacy=0x7ff9880cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.753 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff987400640 1 -- 172.21.15.151:0/2331771195 >> v1:172.21.15.28:6789/0 conn(0x7ff9880d0360 legacy=0x7ff98806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.754 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.749+0000 7ff987400640 1 -- 172.21.15.151:0/2331771195 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98806d8d0 con 0x7ff9880cd4f0 2024-03-21T08:40:44.757 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.753+0000 7ff987400640 1 --2- 172.21.15.151:0/2331771195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880cd4f0 0x7ff988168f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff980009d90 tx=0x7ff98001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f21c807a935011 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9800b2070 con 0x7ff9880cd4f0 2024-03-21T08:40:44.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9800a5940 con 0x7ff9880cd4f0 2024-03-21T08:40:44.758 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff985600640 1 -- 172.21.15.151:0/2331771195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9800a5d10 con 0x7ff9880cd4f0 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 -- 172.21.15.151:0/2331771195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880cd4f0 msgr2=0x7ff988168f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 --2- 172.21.15.151:0/2331771195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880cd4f0 0x7ff988168f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff980009d90 tx=0x7ff98001c970 comp rx=0 tx=0).stop 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 -- 172.21.15.151:0/2331771195 shutdown_connections 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 --2- 172.21.15.151:0/2331771195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880cd4f0 0x7ff988168f30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 -- 172.21.15.151:0/2331771195 >> 172.21.15.151:0/2331771195 conn(0x7ff9880d1ab0 msgr2=0x7ff9880d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 -- 172.21.15.151:0/2331771195 shutdown_connections 2024-03-21T08:40:44.759 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 -- 172.21.15.151:0/2331771195 wait complete. 2024-03-21T08:40:44.761 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.757+0000 7ff98da00640 1 Processor -- start 2024-03-21T08:40:44.762 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- start start 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 0x7ff9880c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 0x7ff9881521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98816c020 con 0x7ff9880d0360 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98816bea0 con 0x7ff9880cd4f0 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98816c1a0 con 0x7ff98815e1e0 2024-03-21T08:40:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff986a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 0x7ff9880c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:45500/0 (socket says 172.21.15.151:45500) 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 -- 172.21.15.151:0/3852232182 learned_addr learned my addr 172.21.15.151:0/3852232182 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff986000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 0x7ff9881521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 -- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 msgr2=0x7ff9880c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 0x7ff9880c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 -- 172.21.15.151:0/3852232182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 msgr2=0x7ff9881521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 0x7ff9881521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 -- 172.21.15.151:0/3852232182 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988154320 con 0x7ff98815e1e0 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff986a00640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 0x7ff9880c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff986000640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 0x7ff9881521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:44.765 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff987400640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff98001a5a0 tx=0x7ff980019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9800b2050 con 0x7ff98815e1e0 2024-03-21T08:40:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9800be070 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff988154600 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980009450 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff988151c90 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff9800adb80 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.761+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff9800c3d20 con 0x7ff98815e1e0 2024-03-21T08:40:44.767 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.765+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9881547d0 con 0x7ff98815e1e0 2024-03-21T08:40:44.768 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.765+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff9881547d0 con 0x7ff98815e1e0 2024-03-21T08:40:44.834 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.833+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a89375db-4a99-4718-b176-616941009b89", "id": 6} v 0) v1 -- 0x7ff98804f4f0 con 0x7ff98815e1e0 2024-03-21T08:40:44.894 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.865+0000 7f1b5f09b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T08:40:44.894 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff97ea00640 1 -- 172.21.15.151:0/3852232182 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a89375db-4a99-4718-b176-616941009b89", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff980016560 con 0x7ff98815e1e0 2024-03-21T08:40:44.894 INFO:teuthology.orchestra.run.smithi151.stdout:6 2024-03-21T08:40:44.895 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.837+0000 7f0c3969b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 msgr2=0x7ff9881539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff98001a5a0 tx=0x7ff980019710 comp rx=0 tx=0).stop 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 shutdown_connections 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff98815e1e0 0x7ff9881539f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9880d0360 0x7ff9881521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 --2- 172.21.15.151:0/3852232182 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff9880cd4f0 0x7ff9880c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 >> 172.21.15.151:0/3852232182 conn(0x7ff9880d1ab0 msgr2=0x7ff9880b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:44.896 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 shutdown_connections 2024-03-21T08:40:44.897 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:44.837+0000 7ff98da00640 1 -- 172.21.15.151:0/3852232182 wait complete. 2024-03-21T08:40:44.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0 7 2024-03-21T08:40:44.912 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:44.909+0000 7f0c3969b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T08:40:44.943 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:44.941+0000 7f1b5f09b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T08:40:45.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.029+0000 7f7a3c400640 1 Processor -- start 2024-03-21T08:40:45.034 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.029+0000 7f7a3c400640 1 -- start start 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3c400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7a340d0360 0x7f7a3406c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3c400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a340b3010 con 0x7f7a3415e1e0 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3c400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3406d310 con 0x7f7a340d0360 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3406d5f0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7a340d0360 0x7f7a3406c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.151:45510/0 (socket says 172.21.15.151:45510) 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7a340d0360 0x7f7a3406c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55940/0 (socket says 172.21.15.151:55940) 2024-03-21T08:40:45.035 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3a400640 1 -- 172.21.15.151:0/4294860287 learned_addr learned my addr 172.21.15.151:0/4294860287 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:45.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 -- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 msgr2=0x7f7a340cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:45.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 -- 172.21.15.151:0/4294860287 >> v1:172.21.15.151:6789/0 conn(0x7f7a3415e1e0 legacy=0x7f7a340cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:45.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a39000640 1 -- 172.21.15.151:0/4294860287 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3406d8d0 con 0x7f7a340d0360 2024-03-21T08:40:45.036 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.033+0000 7f7a3a400640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:45.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a39000640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a340d0360 0x7f7a3406c6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097c0 tx=0x7f7a2001cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce06b6ebfd8f080a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:45.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a33e00640 1 -- 172.21.15.151:0/4294860287 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200b1070 con 0x7f7a340d0360 2024-03-21T08:40:45.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a33e00640 1 -- 172.21.15.151:0/4294860287 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a200ac410 con 0x7f7a340d0360 2024-03-21T08:40:45.038 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a33e00640 1 -- 172.21.15.151:0/4294860287 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200ac7e0 con 0x7f7a340d0360 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a340d0360 msgr2=0x7f7a3406c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a340d0360 0x7f7a3406c6a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097c0 tx=0x7f7a2001cf00 comp rx=0 tx=0).stop 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- 172.21.15.151:0/4294860287 shutdown_connections 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a3406cbe0 0x7f7a340cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 --2- 172.21.15.151:0/4294860287 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a340d0360 0x7f7a3406c6a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- 172.21.15.151:0/4294860287 >> 172.21.15.151:0/4294860287 conn(0x7f7a340d1ab0 msgr2=0x7f7a340d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:45.039 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- 172.21.15.151:0/4294860287 shutdown_connections 2024-03-21T08:40:45.040 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- 172.21.15.151:0/4294860287 wait complete. 2024-03-21T08:40:45.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 Processor -- start 2024-03-21T08:40:45.041 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.037+0000 7f7a3c400640 1 -- start start 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a340d0360 0x7f7a34150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 0x7f7a3414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd600 con 0x7f7a3415e1e0 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd480 con 0x7f7a3406cbe0 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd780 con 0x7f7a340d0360 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a340d0360 0x7f7a34150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.151:55946/0 (socket says 172.21.15.151:55946) 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 -- 172.21.15.151:0/1448417398 learned_addr learned my addr 172.21.15.151:0/1448417398 (peer_addr_for_me v2:172.21.15.151:0/0) 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3a400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 0x7f7a3414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 -- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a340d0360 msgr2=0x7f7a34150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a340d0360 0x7f7a34150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.044 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 -- 172.21.15.151:0/1448417398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 msgr2=0x7f7a3414eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 0x7f7a3414eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 -- 172.21.15.151:0/1448417398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a34151d00 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3a400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 0x7f7a3414eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a39a00640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c420 tx=0x7f7a2800c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a280052d0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a28005470 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28007260 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a34151fe0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a341525a0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7a28007400 con 0x7f7a3406cbe0 2024-03-21T08:40:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7a28003a10 con 0x7f7a3406cbe0 2024-03-21T08:40:45.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.041+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a34152970 con 0x7f7a3406cbe0 2024-03-21T08:40:45.046 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.045+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7a28003600 con 0x7f7a3406cbe0 2024-03-21T08:40:45.112 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.109+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd new", "uuid": "35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0", "id": 7} v 0) v1 -- 0x7f7a3404f4f0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.121 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.117+0000 7f7a31600640 1 -- 172.21.15.151:0/1448417398 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7a28003cf0 con 0x7f7a3406cbe0 2024-03-21T08:40:45.121 INFO:teuthology.orchestra.run.smithi151.stdout:7 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 msgr2=0x7f7a340b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c420 tx=0x7f7a2800c8e0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 shutdown_connections 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a3415e1e0 0x7f7a3414eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a340d0360 0x7f7a34150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 --2- 172.21.15.151:0/1448417398 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a3406cbe0 0x7f7a340b6600 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 >> 172.21.15.151:0/1448417398 conn(0x7f7a340d1ab0 msgr2=0x7f7a34071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 shutdown_connections 2024-03-21T08:40:45.123 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:40:45.121+0000 7f7a3c400640 1 -- 172.21.15.151:0/1448417398 wait complete. 2024-03-21T08:40:45.140 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-21T08:40:45.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-21T08:40:45.143 INFO:tasks.ceph.osd.0:Started 2024-03-21T08:40:45.143 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-21T08:40:45.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-21T08:40:45.146 INFO:tasks.ceph.osd.1:Started 2024-03-21T08:40:45.146 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-21T08:40:45.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-21T08:40:45.149 INFO:tasks.ceph.osd.2:Started 2024-03-21T08:40:45.149 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-21T08:40:45.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-21T08:40:45.152 INFO:tasks.ceph.osd.3:Started 2024-03-21T08:40:45.153 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-21T08:40:45.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-21T08:40:45.189 INFO:tasks.ceph.osd.4:Started 2024-03-21T08:40:45.189 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-21T08:40:45.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-21T08:40:45.193 INFO:tasks.ceph.osd.5:Started 2024-03-21T08:40:45.193 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-21T08:40:45.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-21T08:40:45.197 INFO:tasks.ceph.osd.6:Started 2024-03-21T08:40:45.197 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-21T08:40:45.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-21T08:40:45.202 INFO:tasks.ceph.osd.7:Started 2024-03-21T08:40:45.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:40:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 Processor -- start 2024-03-21T08:40:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 -- start start 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 0x7fcad006cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00cc1d0 con 0x7fcad0099fe0 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad006d450 con 0x7fcad0099b80 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.385+0000 7fcad7000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00d1730 con 0x7fcad015e0c0 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcad5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcad0099fe0 0x7fcad00cd380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59378/0 (socket says 172.21.15.28:59378) 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcad5000640 1 -- 172.21.15.28:0/3930793474 learned_addr learned my addr 172.21.15.28:0/3930793474 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:40:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacfe00640 1 --2- 172.21.15.28:0/3930793474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 0x7fcad006cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacfe00640 1 -- 172.21.15.28:0/3930793474 >> v1:172.21.15.151:6789/0 conn(0x7fcad0099b80 legacy=0x7fcad00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacfe00640 1 -- 172.21.15.28:0/3930793474 >> v1:172.21.15.28:6789/0 conn(0x7fcad0099fe0 legacy=0x7fcad00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacf400640 1 -- 172.21.15.28:0/3930793474 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229837700 0 0) 0x7fcad00cc1d0 con 0x7fcad0099fe0 2024-03-21T08:40:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacfe00640 1 -- 172.21.15.28:0/3930793474 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00d1450 con 0x7fcad015e0c0 2024-03-21T08:40:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacfe00640 1 --2- 172.21.15.28:0/3930793474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 0x7fcad006cf10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009770 tx=0x7fcac001c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6384fe2bd57c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacf400640 1 -- 172.21.15.28:0/3930793474 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00b0070 con 0x7fcad015e0c0 2024-03-21T08:40:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacf400640 1 -- 172.21.15.28:0/3930793474 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00ab600 con 0x7fcad015e0c0 2024-03-21T08:40:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.389+0000 7fcacf400640 1 -- 172.21.15.28:0/3930793474 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00ab9d0 con 0x7fcad015e0c0 2024-03-21T08:40:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- 172.21.15.28:0/3930793474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 msgr2=0x7fcad006cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 --2- 172.21.15.28:0/3930793474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 0x7fcad006cf10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009770 tx=0x7fcac001c310 comp rx=0 tx=0).stop 2024-03-21T08:40:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- 172.21.15.28:0/3930793474 shutdown_connections 2024-03-21T08:40:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 --2- 172.21.15.28:0/3930793474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad015e0c0 0x7fcad006cf10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- 172.21.15.28:0/3930793474 >> 172.21.15.28:0/3930793474 conn(0x7fcad00b34d0 msgr2=0x7fcad00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- 172.21.15.28:0/3930793474 shutdown_connections 2024-03-21T08:40:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- 172.21.15.28:0/3930793474 wait complete. 2024-03-21T08:40:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 Processor -- start 2024-03-21T08:40:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- start start 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad0099fe0 0x7fcad01f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 0x7fcad015e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad009a470 con 0x7fcad015e0c0 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00cc750 con 0x7fcad0099b80 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad7000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad015e4a0 con 0x7fcad0099fe0 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 0x7fcad015e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35028/0 (socket says 172.21.15.28:35028) 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 -- 172.21.15.28:0/2822263409 learned_addr learned my addr 172.21.15.28:0/2822263409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcac7e00640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad0099fe0 0x7fcad01f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 -- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad0099fe0 msgr2=0x7fcad01f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad0099fe0 0x7fcad01f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 -- 172.21.15.28:0/2822263409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 msgr2=0x7fcad015e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 0x7fcad015e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcacfe00640 1 -- 172.21.15.28:0/2822263409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad015ec80 con 0x7fcad0099b80 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.393+0000 7fcad5000640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 0x7fcad015e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:40:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcacfe00640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcac001a9c0 tx=0x7fcac00190b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00b0040 con 0x7fcad0099b80 2024-03-21T08:40:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00bd070 con 0x7fcad0099b80 2024-03-21T08:40:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00b5b70 con 0x7fcad0099b80 2024-03-21T08:40:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad015ef50 con 0x7fcad0099b80 2024-03-21T08:40:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad015f510 con 0x7fcad0099b80 2024-03-21T08:40:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fcac00b3070 con 0x7fcad0099b80 2024-03-21T08:40:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fcac00ab580 con 0x7fcad0099b80 2024-03-21T08:40:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.397+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad004f410 con 0x7fcad0099b80 2024-03-21T08:40:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.401+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fcac0015f90 con 0x7fcad0099b80 2024-03-21T08:40:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.485+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcad00b77c0 con 0x7fcad0099b80 2024-03-21T08:40:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.497+0000 7fcaccc00640 1 -- 172.21.15.28:0/2822263409 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fcac00c2460 con 0x7fcad0099b80 2024-03-21T08:40:45.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:40:45.501 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:45.116957+0000","last_up_change":"0.000000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 msgr2=0x7fcad0152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcac001a9c0 tx=0x7fcac00190b0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 shutdown_connections 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad015e0c0 0x7fcad015e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcad0099fe0 0x7fcad01f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 --2- 172.21.15.28:0/2822263409 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcad0099b80 0x7fcad0152a30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 >> 172.21.15.28:0/2822263409 conn(0x7fcad00b34d0 msgr2=0x7fcad00702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 shutdown_connections 2024-03-21T08:40:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:40:45.501+0000 7fcad7000640 1 -- 172.21.15.28:0/2822263409 wait complete. 2024-03-21T08:40:45.520 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-21T08:40:45.520 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-21T08:40:45.615 INFO:tasks.ceph.mgr.x.smithi151.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-21T08:40:45.615 INFO:tasks.ceph.mgr.x.smithi151.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-21T08:40:45.616 INFO:tasks.ceph.mgr.x.smithi151.stderr: from numpy import show_config as show_numpy_config 2024-03-21T08:40:45.631 INFO:tasks.ceph.mgr.y.smithi028.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-21T08:40:45.631 INFO:tasks.ceph.mgr.y.smithi028.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-21T08:40:45.631 INFO:tasks.ceph.mgr.y.smithi028.stderr: from numpy import show_config as show_numpy_config 2024-03-21T08:40:45.662 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:45.657+0000 7f0c3969b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T08:40:45.678 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:45.677+0000 7f1b5f09b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T08:40:45.798 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-03-21T08:40:45.797+0000 7fef719d37c0 -1 Falling back to public interface 2024-03-21T08:40:45.802 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-03-21T08:40:45.801+0000 7f06db32d7c0 -1 Falling back to public interface 2024-03-21T08:40:45.806 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-03-21T08:40:45.805+0000 7faab0a547c0 -1 Falling back to public interface 2024-03-21T08:40:45.806 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-03-21T08:40:45.805+0000 7fb84a0d07c0 -1 Falling back to public interface 2024-03-21T08:40:45.808 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:45.805+0000 7f0c3969b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T08:40:45.839 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:45.837+0000 7f1b5f09b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T08:40:45.867 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-21T08:40:45.865+0000 7f51aaedb7c0 -1 Falling back to public interface 2024-03-21T08:40:45.883 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-21T08:40:45.881+0000 7f7bc17887c0 -1 Falling back to public interface 2024-03-21T08:40:45.883 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-21T08:40:45.881+0000 7f4c909a97c0 -1 Falling back to public interface 2024-03-21T08:40:45.899 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-21T08:40:45.897+0000 7fe71143d7c0 -1 Falling back to public interface 2024-03-21T08:40:45.997 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:45.993+0000 7f0c3969b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T08:40:46.028 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:46.025+0000 7f1b5f09b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T08:40:46.070 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:46.069+0000 7f0c3969b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T08:40:46.105 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:46.105+0000 7f1b5f09b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T08:40:46.220 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:46.217+0000 7f0c3969b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T08:40:46.265 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:46.261+0000 7f1b5f09b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T08:40:46.326 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-21T08:40:46.460 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 Processor -- start 2024-03-21T08:40:46.460 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 -- start start 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a80b15f0 0x7fb9a806b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9a815c9b0 0x7fb9a815f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a806ba90 con 0x7fb9a806b170 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a80b1bb0 con 0x7fb9a815c9b0 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.457+0000 7fb9af000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80b1e90 con 0x7fb9a80b15f0 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9a7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9a806b170 0x7fb9a80d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59386/0 (socket says 172.21.15.28:59386) 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9a7e00640 1 -- 172.21.15.28:0/1872364691 learned_addr learned my addr 172.21.15.28:0/1872364691 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:40:46.461 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 --2- 172.21.15.28:0/1872364691 >> v2:172.21.15.28:3300/0 conn(0x7fb9a815c9b0 0x7fb9a815f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:46.462 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb99fe00640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a80b15f0 0x7fb9a806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:46.462 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 -- 172.21.15.28:0/1872364691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a80b15f0 msgr2=0x7fb9a806b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:46.462 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a80b15f0 0x7fb9a806b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.462 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 -- 172.21.15.28:0/1872364691 >> v1:172.21.15.28:6789/0 conn(0x7fb9a806b170 legacy=0x7fb9a80d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:46.462 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 -- 172.21.15.28:0/1872364691 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80b2170 con 0x7fb9a815c9b0 2024-03-21T08:40:46.464 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9ad000640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a815c9b0 0x7fb9a815f730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009d90 tx=0x7fb9a001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76aa47596fe73cb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:46.465 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9a7400640 1 -- 172.21.15.28:0/1872364691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a00b2070 con 0x7fb9a815c9b0 2024-03-21T08:40:46.465 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9a7400640 1 -- 172.21.15.28:0/1872364691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a00a5910 con 0x7fb9a815c9b0 2024-03-21T08:40:46.465 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.461+0000 7fb9a7400640 1 -- 172.21.15.28:0/1872364691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a00a5ce0 con 0x7fb9a815c9b0 2024-03-21T08:40:46.465 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- 172.21.15.28:0/1872364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a815c9b0 msgr2=0x7fb9a815f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a815c9b0 0x7fb9a815f730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009d90 tx=0x7fb9a001c940 comp rx=0 tx=0).stop 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- 172.21.15.28:0/1872364691 shutdown_connections 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a815c9b0 0x7fb9a815f730 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- 172.21.15.28:0/1872364691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a80b15f0 0x7fb9a806b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- 172.21.15.28:0/1872364691 >> 172.21.15.28:0/1872364691 conn(0x7fb9a8099a40 msgr2=0x7fb9a80d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- 172.21.15.28:0/1872364691 shutdown_connections 2024-03-21T08:40:46.466 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- 172.21.15.28:0/1872364691 wait complete. 2024-03-21T08:40:46.467 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 Processor -- start 2024-03-21T08:40:46.467 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- start start 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9a80b15f0 0x7fb9a80af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a815c9b0 0x7fb9a80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8159b30 con 0x7fb9a806b170 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a81599b0 con 0x7fb9a80b15f0 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9af000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a81692e0 con 0x7fb9a815c9b0 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50942/0 (socket says 172.21.15.28:50942) 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 -- 172.21.15.28:0/3837101547 learned_addr learned my addr 172.21.15.28:0/3837101547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9ad000640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a815c9b0 0x7fb9a80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb99fe00640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9a80b15f0 0x7fb9a80af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:46.469 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 -- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a815c9b0 msgr2=0x7fb9a80aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:46.470 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a815c9b0 0x7fb9a80aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.470 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 -- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9a80b15f0 msgr2=0x7fb9a80af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:46.470 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9a80b15f0 0x7fb9a80af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.470 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.465+0000 7fb9a7e00640 1 -- 172.21.15.28:0/3837101547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a81f71f0 con 0x7fb9a806b170 2024-03-21T08:40:46.470 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a7e00640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c370 tx=0x7fb99800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998017070 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb998005cf0 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9980072f0 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a81f74d0 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a81f7a90 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb99801a070 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb998007490 con 0x7fb9a806b170 2024-03-21T08:40:46.471 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.469+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a8074b20 con 0x7fb9a806b170 2024-03-21T08:40:46.482 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.481+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb998021020 con 0x7fb9a806b170 2024-03-21T08:40:46.541 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9a806c000 con 0x7fb9a806b170 2024-03-21T08:40:46.542 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9a4c00640 1 -- 172.21.15.28:0/3837101547 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb99801c440 con 0x7fb9a806b170 2024-03-21T08:40:46.542 INFO:teuthology.misc.health.smithi028.stdout: 2024-03-21T08:40:46.542 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:45.116957+0000","last_up_change":"0.000000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:40:46.543 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 msgr2=0x7fb9a8155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:46.543 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c370 tx=0x7fb99800c830 comp rx=0 tx=0).stop 2024-03-21T08:40:46.543 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 shutdown_connections 2024-03-21T08:40:46.543 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9a815c9b0 0x7fb9a80aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.544 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9a80b15f0 0x7fb9a80af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.544 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 --2- 172.21.15.28:0/3837101547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9a806b170 0x7fb9a8155570 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:46.544 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 >> 172.21.15.28:0/3837101547 conn(0x7fb9a8099a40 msgr2=0x7fb9a80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:46.544 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 shutdown_connections 2024-03-21T08:40:46.544 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:46.541+0000 7fb9af000640 1 -- 172.21.15.28:0/3837101547 wait complete. 2024-03-21T08:40:46.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-21T08:40:46.927 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:46.925+0000 7f0c3969b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T08:40:46.996 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:46.993+0000 7f1b5f09b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T08:40:47.013 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:47.009+0000 7f0c3969b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T08:40:47.090 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:47.089+0000 7f1b5f09b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T08:40:47.216 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:47.213+0000 7f0c3969b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T08:40:47.301 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:47.297+0000 7f0c3969b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T08:40:47.302 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:47.301+0000 7f1b5f09b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T08:40:47.393 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:47.389+0000 7f1b5f09b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T08:40:47.631 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:47.629+0000 7f0c3969b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T08:40:47.702 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:47.701+0000 7f0c3969b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T08:40:47.743 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:47.741+0000 7f1b5f09b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T08:40:47.820 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-03-21T08:40:47.817+0000 7f1b5f09b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T08:40:48.724 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:48.721+0000 7f0bdf800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-21T08:40:48.725 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-03-21T08:40:48.721+0000 7f0bdf800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-21T08:40:49.633 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-21T08:40:49.629+0000 7f51aaedb7c0 -1 osd.5 0 log_to_monitors true 2024-03-21T08:40:49.703 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-21T08:40:49.701+0000 7f4c909a97c0 -1 osd.4 0 log_to_monitors true 2024-03-21T08:40:49.748 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-03-21T08:40:49.745+0000 7f06db32d7c0 -1 osd.0 0 log_to_monitors true 2024-03-21T08:40:49.749 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-03-21T08:40:49.745+0000 7fb84a0d07c0 -1 osd.3 0 log_to_monitors true 2024-03-21T08:40:49.823 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-21T08:40:49.821+0000 7f7bc17887c0 -1 osd.6 0 log_to_monitors true 2024-03-21T08:40:49.916 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-03-21T08:40:49.913+0000 7faab0a547c0 -1 osd.2 0 log_to_monitors true 2024-03-21T08:40:49.920 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-03-21T08:40:49.917+0000 7fef719d37c0 -1 osd.1 0 log_to_monitors true 2024-03-21T08:40:49.927 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-21T08:40:49.925+0000 7fe71143d7c0 -1 osd.7 0 log_to_monitors true 2024-03-21T08:40:51.853 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-03-21T08:40:51.848+0000 7fb845000640 -1 osd.3 0 waiting for initial osdmap 2024-03-21T08:40:51.874 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-03-21T08:40:51.876+0000 7f06d6200640 -1 osd.0 0 waiting for initial osdmap 2024-03-21T08:40:51.885 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-03-21T08:40:51.884+0000 7fb83b000640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:51.900 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-03-21T08:40:51.900+0000 7f06cc200640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:52.219 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-21T08:40:52.216+0000 7f4c8b800640 -1 osd.4 0 waiting for initial osdmap 2024-03-21T08:40:52.279 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-03-21T08:40:52.276+0000 7f4c81800640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:52.314 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-21T08:40:52.308+0000 7f51a5e00640 -1 osd.5 0 waiting for initial osdmap 2024-03-21T08:40:52.413 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-03-21T08:40:52.408+0000 7f519be00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:52.863 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-03-21T08:40:52.864+0000 7faaaba00640 -1 osd.2 0 waiting for initial osdmap 2024-03-21T08:40:52.891 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-03-21T08:40:52.892+0000 7faaa1a00640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:52.943 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-03-21T08:40:52.944+0000 7fef6c800640 -1 osd.1 0 waiting for initial osdmap 2024-03-21T08:40:52.963 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-03-21T08:40:52.964+0000 7fef62800640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:53.206 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-21T08:40:53.200+0000 7fe70c400640 -1 osd.7 0 waiting for initial osdmap 2024-03-21T08:40:53.224 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-21T08:40:53.220+0000 7f7bbd000640 -1 osd.6 0 waiting for initial osdmap 2024-03-21T08:40:53.231 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-03-21T08:40:53.228+0000 7fe702400640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:53.263 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-03-21T08:40:53.260+0000 7f7bb2600640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T08:40:53.366 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-21T08:40:53.498 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 Processor -- start 2024-03-21T08:40:53.498 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 -- start start 2024-03-21T08:40:53.500 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f319015dc40 0x7f31900cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.500 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f3190099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.500 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 0x7f31900cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.500 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31900cda00 con 0x7f319015dc40 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31900cdce0 con 0x7f3190099f80 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31900cdfc0 con 0x7f319009aa10 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f319015dc40 0x7f31900cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f319015dc40 0x7f31900cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48470/0 (socket says 172.21.15.28:48470) 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 -- 172.21.15.28:0/3210518888 learned_addr learned my addr 172.21.15.28:0/3210518888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318e000640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f3190099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318f400640 1 --2- 172.21.15.28:0/3210518888 >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 0x7f31900cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 -- 172.21.15.28:0/3210518888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 msgr2=0x7f3190099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f3190099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.501 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 -- 172.21.15.28:0/3210518888 >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 msgr2=0x7f31900cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:40:53.502 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 --2- 172.21.15.28:0/3210518888 >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 0x7f31900cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.502 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318ea00640 1 -- 172.21.15.28:0/3210518888 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319006c280 con 0x7f319015dc40 2024-03-21T08:40:53.502 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.500+0000 7f318f400640 1 --2- 172.21.15.28:0/3210518888 >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 0x7f31900cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:40:53.503 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318ea00640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f319015dc40 0x7f31900cd120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f31900b78d0 tx=0x7f318001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780c96b6a4c4f949 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:53.503 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318d600640 1 -- 172.21.15.28:0/3210518888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31800b0070 con 0x7f319015dc40 2024-03-21T08:40:53.503 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318d600640 1 -- 172.21.15.28:0/3210518888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31800ab600 con 0x7f319015dc40 2024-03-21T08:40:53.503 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318d600640 1 -- 172.21.15.28:0/3210518888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31800ab9d0 con 0x7f319015dc40 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- 172.21.15.28:0/3210518888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f319015dc40 msgr2=0x7f31900cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f319015dc40 0x7f31900cd120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f31900b78d0 tx=0x7f318001c260 comp rx=0 tx=0).stop 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- 172.21.15.28:0/3210518888 shutdown_connections 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 --2- 172.21.15.28:0/3210518888 >> v2:172.21.15.151:3300/0 conn(0x7f3190099f80 0x7f31900cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f3190099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.504 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 --2- 172.21.15.28:0/3210518888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f319015dc40 0x7f31900cd120 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.505 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- 172.21.15.28:0/3210518888 >> 172.21.15.28:0/3210518888 conn(0x7f31900b3310 msgr2=0x7f31900b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:53.505 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- 172.21.15.28:0/3210518888 shutdown_connections 2024-03-21T08:40:53.505 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- 172.21.15.28:0/3210518888 wait complete. 2024-03-21T08:40:53.506 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 Processor -- start 2024-03-21T08:40:53.506 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.504+0000 7f318fe00640 1 -- start start 2024-03-21T08:40:53.508 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.508 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f319014c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.508 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f319015dc40 0x7f31901fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.508 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319006cc00 con 0x7f3190099f80 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319006ca80 con 0x7f319015dc40 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319006cd80 con 0x7f319009aa10 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48480/0 (socket says 172.21.15.28:48480) 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 -- 172.21.15.28:0/1054935336 learned_addr learned my addr 172.21.15.28:0/1054935336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:40:53.509 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318e000640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f319014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318f400640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f319015dc40 0x7f31901fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 -- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 msgr2=0x7f319014c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f319014c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 -- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f319015dc40 msgr2=0x7f31901fa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f319015dc40 0x7f31901fa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 -- 172.21.15.28:0/1054935336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190155a40 con 0x7f3190099f80 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318ea00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f31800ae040 tx=0x7f3180019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31800b0040 con 0x7f3190099f80 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31800b9070 con 0x7f3190099f80 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31800b4660 con 0x7f3190099f80 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3190155d20 con 0x7f3190099f80 2024-03-21T08:40:53.510 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319015b2f0 con 0x7f3190099f80 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190074b20 con 0x7f3190099f80 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.508+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 87403+0+0 (secure 0 0 0) 0x7f31800b4800 con 0x7f3190099f80 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.512+0000 7f3187400640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 0x7f3170069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.512+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3351+0+0 (secure 0 0 0) 0x7f318011f1c0 con 0x7f3190099f80 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.512+0000 7f318e000640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 0x7f3170069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:40:53.512 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.512+0000 7f318e000640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 0x7f3170069f50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3190150e80 tx=0x7f317c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:40:53.515 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.516+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3180016110 con 0x7f3190099f80 2024-03-21T08:40:53.680 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.680+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31900af260 con 0x7f3190099f80 2024-03-21T08:40:53.681 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.680+0000 7f3187400640 1 -- 172.21.15.28:0/1054935336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9001 (secure 0 0 0) 0x7f31800f0540 con 0x7f3190099f80 2024-03-21T08:40:53.682 INFO:teuthology.misc.health.smithi028.stdout: 2024-03-21T08:40:53.682 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":14,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:52.887618+0000","last_up_change":"2024-03-21T08:40:52.887618+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:40:53.684 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 msgr2=0x7f3170069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:53.684 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 0x7f3170069f50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3190150e80 tx=0x7f317c008040 comp rx=0 tx=0).stop 2024-03-21T08:40:53.684 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 msgr2=0x7f31900b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:40:53.684 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f31800ae040 tx=0x7f3180019000 comp rx=0 tx=0).stop 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 shutdown_connections 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f319015dc40 0x7f31901fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3170067a90 0x7f3170069f50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f319009aa10 0x7f319014c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 --2- 172.21.15.28:0/1054935336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3190099f80 0x7f31900b1b00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:40:53.685 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 >> 172.21.15.28:0/1054935336 conn(0x7f31900b3310 msgr2=0x7f31901571f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:40:53.686 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 shutdown_connections 2024-03-21T08:40:53.686 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:40:53.684+0000 7f318fe00640 1 -- 172.21.15.28:0/1054935336 wait complete. 2024-03-21T08:40:53.700 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-03-21T08:41:00.506 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:00.639 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 Processor -- start 2024-03-21T08:41:00.639 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 -- start start 2024-03-21T08:41:00.640 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb07815daf0 0x7fb0781616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0780b2370 0x7fb0780b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb078162640 con 0x7fb07806bb80 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0780b2d20 con 0x7fb07815daf0 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07e400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07806c010 con 0x7fb0780b2370 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb07815daf0 0x7fb0781616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb07815daf0 0x7fb0781616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35280/0 (socket says 172.21.15.28:35280) 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 -- 172.21.15.28:0/3716116029 learned_addr learned my addr 172.21.15.28:0/3716116029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 -- 172.21.15.28:0/3716116029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0780b2370 msgr2=0x7fb0780b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 --2- 172.21.15.28:0/3716116029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0780b2370 0x7fb0780b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.641 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 -- 172.21.15.28:0/3716116029 >> v1:172.21.15.28:6789/0 conn(0x7fb07806bb80 legacy=0x7fb078070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.642 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.639+0000 7fb07d000640 1 -- 172.21.15.28:0/3716116029 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078070770 con 0x7fb07815daf0 2024-03-21T08:41:00.643 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07d000640 1 --2- 172.21.15.28:0/3716116029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815daf0 0x7fb0781616e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009870 tx=0x7fb06c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c8820992843f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:00.644 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb077400640 1 -- 172.21.15.28:0/3716116029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0a95f0 con 0x7fb07815daf0 2024-03-21T08:41:00.644 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb077400640 1 -- 172.21.15.28:0/3716116029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c0a9790 con 0x7fb07815daf0 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb077400640 1 -- 172.21.15.28:0/3716116029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0a9b60 con 0x7fb07815daf0 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- 172.21.15.28:0/3716116029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815daf0 msgr2=0x7fb0781616e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 --2- 172.21.15.28:0/3716116029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815daf0 0x7fb0781616e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009870 tx=0x7fb06c01c530 comp rx=0 tx=0).stop 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- 172.21.15.28:0/3716116029 shutdown_connections 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 --2- 172.21.15.28:0/3716116029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0780b2370 0x7fb0780b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 --2- 172.21.15.28:0/3716116029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815daf0 0x7fb0781616e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- 172.21.15.28:0/3716116029 >> 172.21.15.28:0/3716116029 conn(0x7fb0780d0710 msgr2=0x7fb0780d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- 172.21.15.28:0/3716116029 shutdown_connections 2024-03-21T08:41:00.645 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- 172.21.15.28:0/3716116029 wait complete. 2024-03-21T08:41:00.646 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 Processor -- start 2024-03-21T08:41:00.646 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.643+0000 7fb07e400640 1 -- start start 2024-03-21T08:41:00.648 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07806bb80 0x7fb0780b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.648 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07815daf0 0x7fb07814d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07815ded0 con 0x7fb0780b2370 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780c9b70 con 0x7fb07806bb80 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb078154420 con 0x7fb07815daf0 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35294/0 (socket says 172.21.15.28:35294) 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 -- 172.21.15.28:0/601732816 learned_addr learned my addr 172.21.15.28:0/601732816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07da00640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07815daf0 0x7fb07814d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:00.649 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07d000640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07806bb80 0x7fb0780b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 -- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07815daf0 msgr2=0x7fb07814d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07815daf0 0x7fb07814d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 -- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07806bb80 msgr2=0x7fb0780b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07806bb80 0x7fb0780b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 -- 172.21.15.28:0/601732816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07814e420 con 0x7fb0780b2370 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb077e00640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c420 tx=0x7fb06400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640052d0 con 0x7fb0780b2370 2024-03-21T08:41:00.650 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064005470 con 0x7fb0780b2370 2024-03-21T08:41:00.651 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064007260 con 0x7fb0780b2370 2024-03-21T08:41:00.651 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07814e700 con 0x7fb0780b2370 2024-03-21T08:41:00.651 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0781549c0 con 0x7fb0780b2370 2024-03-21T08:41:00.651 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb078074b20 con 0x7fb0780b2370 2024-03-21T08:41:00.652 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.647+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb064007400 con 0x7fb0780b2370 2024-03-21T08:41:00.652 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.651+0000 7fb074c00640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 0x7fb05806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:00.652 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.651+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0640850e0 con 0x7fb0780b2370 2024-03-21T08:41:00.652 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.651+0000 7fb07d000640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 0x7fb05806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:00.652 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.651+0000 7fb07d000640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 0x7fb05806a010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0ae000 tx=0x7fb06c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:00.656 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.655+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb064055e30 con 0x7fb0780b2370 2024-03-21T08:41:00.822 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb078154b90 con 0x7fb0780b2370 2024-03-21T08:41:00.823 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb074c00640 1 -- 172.21.15.28:0/601732816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fb064055730 con 0x7fb0780b2370 2024-03-21T08:41:00.824 INFO:teuthology.misc.health.smithi028.stdout: 2024-03-21T08:41:00.824 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":17,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:55.899749+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 msgr2=0x7fb05806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 0x7fb05806a010 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0ae000 tx=0x7fb06c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 msgr2=0x7fb0780b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb06400c420 tx=0x7fb06400c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 shutdown_connections 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b50 0x7fb05806a010 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07815daf0 0x7fb07814d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780b2370 0x7fb0780b0e30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 --2- 172.21.15.28:0/601732816 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07806bb80 0x7fb0780b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 >> 172.21.15.28:0/601732816 conn(0x7fb0780d0710 msgr2=0x7fb07815e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 shutdown_connections 2024-03-21T08:41:00.826 INFO:teuthology.misc.health.smithi028.stderr:2024-03-21T08:41:00.823+0000 7fb07e400640 1 -- 172.21.15.28:0/601732816 wait complete. 2024-03-21T08:41:00.843 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-21T08:41:00.843 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-21T08:41:00.844 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-21T08:41:00.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-21T08:41:00.847 INFO:tasks.ceph.mds.a:Started 2024-03-21T08:41:00.847 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-21T08:41:00.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-21T08:41:00.850 INFO:tasks.ceph.mds.c:Started 2024-03-21T08:41:00.850 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-21T08:41:00.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-21T08:41:00.854 INFO:tasks.ceph.mds.b:Started 2024-03-21T08:41:00.854 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-21T08:41:00.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-21T08:41:00.857 INFO:tasks.ceph.mds.d:Started 2024-03-21T08:41:00.857 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-21T08:41:00.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:00.948 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-03-21T08:41:00.950 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-03-21T08:41:00.963 INFO:tasks.ceph.mds.b.smithi151.stdout:starting mds.b at 2024-03-21T08:41:00.964 INFO:tasks.ceph.mds.d.smithi151.stdout:starting mds.d at 2024-03-21T08:41:01.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 Processor -- start 2024-03-21T08:41:01.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 -- start start 2024-03-21T08:41:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f99f00b5520 0x7f99f015c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f015bf00 con 0x7f99f00b34f0 2024-03-21T08:41:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f00ce6e0 con 0x7f99f00b5520 2024-03-21T08:41:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00cdad0 con 0x7f99f00af060 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f5400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f5e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f99f00b5520 0x7f99f015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f4a00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f99f00b34f0 0x7f99f00b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:33504/0 (socket says 172.21.15.28:33504) 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f5400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:60828/0 (socket says 172.21.15.28:60828) 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.003+0000 7f99f4a00640 1 -- 172.21.15.28:0/835773599 learned_addr learned my addr 172.21.15.28:0/835773599 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99efe00640 1 -- 172.21.15.28:0/835773599 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118601673 0 0) 0x7f99f015bf00 con 0x7f99f00b34f0 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5e00640 1 -- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 msgr2=0x7f99f00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5e00640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5e00640 1 -- 172.21.15.28:0/835773599 >> v1:172.21.15.151:6789/0 conn(0x7f99f00b34f0 legacy=0x7f99f00b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5e00640 1 -- 172.21.15.28:0/835773599 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00cddb0 con 0x7f99f00b5520 2024-03-21T08:41:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5400640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:01.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f5e00640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b5520 0x7f99f015c200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009d90 tx=0x7f99e801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266fd85cd80c8af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99efe00640 1 -- 172.21.15.28:0/835773599 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80b2070 con 0x7f99f00b5520 2024-03-21T08:41:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99efe00640 1 -- 172.21.15.28:0/835773599 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80a5890 con 0x7f99f00b5520 2024-03-21T08:41:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99efe00640 1 -- 172.21.15.28:0/835773599 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80a5c60 con 0x7f99f00b5520 2024-03-21T08:41:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 -- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b5520 msgr2=0x7f99f015c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b5520 0x7f99f015c200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009d90 tx=0x7f99e801c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 -- 172.21.15.28:0/835773599 shutdown_connections 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b5520 0x7f99f015c200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 --2- 172.21.15.28:0/835773599 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00af060 0x7f99f00cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.007+0000 7f99f7e00640 1 -- 172.21.15.28:0/835773599 >> 172.21.15.28:0/835773599 conn(0x7f99f00cc230 msgr2=0x7f99f00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- 172.21.15.28:0/835773599 shutdown_connections 2024-03-21T08:41:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- 172.21.15.28:0/835773599 wait complete. 2024-03-21T08:41:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 Processor -- start 2024-03-21T08:41:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- start start 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b34f0 0x7f99f014c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00b5520 0x7f99f00adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f015d0d0 con 0x7f99f00ae0d0 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f015cf50 con 0x7f99f00b34f0 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f015d250 con 0x7f99f00b5520 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35344/0 (socket says 172.21.15.28:35344) 2024-03-21T08:41:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 -- 172.21.15.28:0/1619309196 learned_addr learned my addr 172.21.15.28:0/1619309196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5400640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b34f0 0x7f99f014c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f4a00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00b5520 0x7f99f00adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 -- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00b5520 msgr2=0x7f99f00adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00b5520 0x7f99f00adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 -- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b34f0 msgr2=0x7f99f014c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b34f0 0x7f99f014c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 -- 172.21.15.28:0/1619309196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00acaf0 con 0x7f99f00ae0d0 2024-03-21T08:41:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.011+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99e801af90 tx=0x7f99e8019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80b2050 con 0x7f99f00ae0d0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80be070 con 0x7f99f00ae0d0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80ad850 con 0x7f99f00ae0d0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f00acdd0 con 0x7f99f00ae0d0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f00ad390 con 0x7f99f00ae0d0 2024-03-21T08:41:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f004f410 con 0x7f99f00ae0d0 2024-03-21T08:41:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f99e80ad9f0 con 0x7f99f00ae0d0 2024-03-21T08:41:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 0x7f99d0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f99e8120e90 con 0x7f99f00ae0d0 2024-03-21T08:41:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.015+0000 7f99f5400640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 0x7f99d0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.019+0000 7f99f5400640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 0x7f99d0069fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009580 tx=0x7f99e4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.023+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f99e8016560 con 0x7f99f00ae0d0 2024-03-21T08:41:01.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.183+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99f00ccd50 con 0x7f99f00ae0d0 2024-03-21T08:41:01.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99ed600640 1 -- 172.21.15.28:0/1619309196 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f99e8122090 con 0x7f99f00ae0d0 2024-03-21T08:41:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:01.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:55.899749+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:01.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 msgr2=0x7f99d0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 0x7f99d0069fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009580 tx=0x7f99e4009190 comp rx=0 tx=0).stop 2024-03-21T08:41:01.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 msgr2=0x7f99f00b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99e801af90 tx=0x7f99e8019680 comp rx=0 tx=0).stop 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 shutdown_connections 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f00ae0d0 0x7f99f00b01b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99d0067b20 0x7f99d0069fe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99f00b5520 0x7f99f00adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 --2- 172.21.15.28:0/1619309196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99f00b34f0 0x7f99f014c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 >> 172.21.15.28:0/1619309196 conn(0x7f99f00cc230 msgr2=0x7f99f00afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 shutdown_connections 2024-03-21T08:41:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.187+0000 7f99f7e00640 1 -- 172.21.15.28:0/1619309196 wait complete. 2024-03-21T08:41:01.208 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:41:01.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 Processor -- start 2024-03-21T08:41:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 -- start start 2024-03-21T08:41:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd0300cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd03006cdb0 0x7fd0300cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0300b3010 con 0x7fd0300d0190 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0300d05f0 con 0x7fd03006cdb0 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd034a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03006d310 con 0x7fd03015e1e0 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd03006cdb0 0x7fd0300cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd03006cdb0 0x7fd0300cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35346/0 (socket says 172.21.15.28:35346) 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 -- 172.21.15.28:0/2198350187 learned_addr learned my addr 172.21.15.28:0/2198350187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02de00640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd0300cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 -- 172.21.15.28:0/2198350187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 msgr2=0x7fd0300cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd0300cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 -- 172.21.15.28:0/2198350187 >> v1:172.21.15.151:6789/0 conn(0x7fd0300d0190 legacy=0x7fd03006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.351+0000 7fd02e800640 1 -- 172.21.15.28:0/2198350187 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03006d5f0 con 0x7fd03006cdb0 2024-03-21T08:41:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd02e800640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300cc4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd028009e50 tx=0x7fd02801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34946cdd26dad3c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd02ca00640 1 -- 172.21.15.28:0/2198350187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0280a57f0 con 0x7fd03006cdb0 2024-03-21T08:41:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd02ca00640 1 -- 172.21.15.28:0/2198350187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0280a5990 con 0x7fd03006cdb0 2024-03-21T08:41:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd02ca00640 1 -- 172.21.15.28:0/2198350187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0280a5d60 con 0x7fd03006cdb0 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- 172.21.15.28:0/2198350187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 msgr2=0x7fd0300cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300cc4e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd028009e50 tx=0x7fd02801ca30 comp rx=0 tx=0).stop 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- 172.21.15.28:0/2198350187 shutdown_connections 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300cc4e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 --2- 172.21.15.28:0/2198350187 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd0300cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- 172.21.15.28:0/2198350187 >> 172.21.15.28:0/2198350187 conn(0x7fd0300d1ab0 msgr2=0x7fd0300d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- 172.21.15.28:0/2198350187 shutdown_connections 2024-03-21T08:41:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- 172.21.15.28:0/2198350187 wait complete. 2024-03-21T08:41:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 Processor -- start 2024-03-21T08:41:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.355+0000 7fd034a00640 1 -- start start 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 0x7fd03015e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd030152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0300cd5c0 con 0x7fd03006cdb0 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0300cd440 con 0x7fd0300d0190 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0300cd740 con 0x7fd03015e1e0 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35348/0 (socket says 172.21.15.28:35348) 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 -- 172.21.15.28:0/1695638761 learned_addr learned my addr 172.21.15.28:0/1695638761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02d400640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 0x7fd03015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 -- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 msgr2=0x7fd030152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02e800640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd030152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd030152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 -- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 msgr2=0x7fd03015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 0x7fd03015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 -- 172.21.15.28:0/1695638761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0301612e0 con 0x7fd03006cdb0 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02d400640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 0x7fd03015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02e800640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd030152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd02de00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd02000e870 tx=0x7fd02000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd020017070 con 0x7fd03006cdb0 2024-03-21T08:41:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02000c940 con 0x7fd03006cdb0 2024-03-21T08:41:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0301615c0 con 0x7fd03006cdb0 2024-03-21T08:41:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02000ccf0 con 0x7fd03006cdb0 2024-03-21T08:41:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd030156980 con 0x7fd03006cdb0 2024-03-21T08:41:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.359+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03004f4f0 con 0x7fd03006cdb0 2024-03-21T08:41:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.363+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd02001a070 con 0x7fd03006cdb0 2024-03-21T08:41:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.363+0000 7fd026000640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 0x7fd010069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.363+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd020085a80 con 0x7fd03006cdb0 2024-03-21T08:41:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.363+0000 7fd02d400640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 0x7fd010069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.363+0000 7fd02d400640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 0x7fd010069fe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd0300b7070 tx=0x7fd01c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.367+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd020087e20 con 0x7fd03006cdb0 2024-03-21T08:41:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0300d2830 con 0x7fd03006cdb0 2024-03-21T08:41:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd026000640 1 -- 172.21.15.28:0/1695638761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0200567d0 con 0x7fd03006cdb0 2024-03-21T08:41:01.533 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 msgr2=0x7fd010069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 0x7fd010069fe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd0300b7070 tx=0x7fd01c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 msgr2=0x7fd0300b72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.531+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd02000e870 tx=0x7fd02000ec70 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 shutdown_connections 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd010067b20 0x7fd010069fe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd03015e1e0 0x7fd030152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0300d0190 0x7fd03015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 --2- 172.21.15.28:0/1695638761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd03006cdb0 0x7fd0300b72f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 >> 172.21.15.28:0/1695638761 conn(0x7fd0300d1ab0 msgr2=0x7fd0300cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 shutdown_connections 2024-03-21T08:41:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.535+0000 7fd034a00640 1 -- 172.21.15.28:0/1695638761 wait complete. 2024-03-21T08:41:01.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:01.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.691+0000 7fd4ba000640 1 Processor -- start 2024-03-21T08:41:01.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.691+0000 7fd4ba000640 1 -- start start 2024-03-21T08:41:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4ba000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd4b415e610 0x7fd4b415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4ba000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 0x7fd4b40b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4ba000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b415ebf0 con 0x7fd4b40ce490 2024-03-21T08:41:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4ba000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b415eed0 con 0x7fd4b415e610 2024-03-21T08:41:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4ba000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b415f1b0 con 0x7fd4b40b4380 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4b3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 0x7fd4b40b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4aaa00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd4b415e610 0x7fd4b415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4b3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 0x7fd4b40b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:60854/0 (socket says 172.21.15.28:60854) 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4b3e00640 1 -- 172.21.15.28:0/3669092466 learned_addr learned my addr 172.21.15.28:0/3669092466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4b3400640 1 --1- 172.21.15.28:0/3669092466 >> v1:172.21.15.151:6789/0 conn(0x7fd4b40ce490 0x7fd4b40ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:33524/0 (socket says 172.21.15.28:33524) 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4aaa00640 1 -- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 msgr2=0x7fd4b40b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 0x7fd4b40b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4aaa00640 1 -- 172.21.15.28:0/3669092466 >> v1:172.21.15.151:6789/0 conn(0x7fd4b40ce490 legacy=0x7fd4b40ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.695+0000 7fd4aaa00640 1 -- 172.21.15.28:0/3669092466 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b415f490 con 0x7fd4b415e610 2024-03-21T08:41:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b415e610 0x7fd4b415ba70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0097c0 tx=0x7fd49c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab371dfb3248d85e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4b2a00640 1 -- 172.21.15.28:0/3669092466 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0b0070 con 0x7fd4b415e610 2024-03-21T08:41:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4b2a00640 1 -- 172.21.15.28:0/3669092466 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c0ab600 con 0x7fd4b415e610 2024-03-21T08:41:01.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4b2a00640 1 -- 172.21.15.28:0/3669092466 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0ab9d0 con 0x7fd4b415e610 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b415e610 msgr2=0x7fd4b415ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 --2- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b415e610 0x7fd4b415ba70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0097c0 tx=0x7fd49c01c410 comp rx=0 tx=0).stop 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- 172.21.15.28:0/3669092466 shutdown_connections 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 --2- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40b4380 0x7fd4b40b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 --2- 172.21.15.28:0/3669092466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b415e610 0x7fd4b415ba70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- 172.21.15.28:0/3669092466 >> 172.21.15.28:0/3669092466 conn(0x7fd4b40cc230 msgr2=0x7fd4b40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- 172.21.15.28:0/3669092466 shutdown_connections 2024-03-21T08:41:01.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- 172.21.15.28:0/3669092466 wait complete. 2024-03-21T08:41:01.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 Processor -- start 2024-03-21T08:41:01.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.699+0000 7fd4ba000640 1 -- start start 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b40b4380 0x7fd4b414c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40ce490 0x7fd4b406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40cd710 con 0x7fd4b415e610 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40cd590 con 0x7fd4b40b4380 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40cd890 con 0x7fd4b40ce490 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35350/0 (socket says 172.21.15.28:35350) 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 -- 172.21.15.28:0/1815941899 learned_addr learned my addr 172.21.15.28:0/1815941899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3400640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b40b4380 0x7fd4b414c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40ce490 0x7fd4b406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 -- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40ce490 msgr2=0x7fd4b406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40ce490 0x7fd4b406e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 -- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b40b4380 msgr2=0x7fd4b414c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b40b4380 0x7fd4b414c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 -- 172.21.15.28:0/1815941899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b415e3e0 con 0x7fd4b415e610 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4b3e00640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e880 tx=0x7fd4ac00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac017070 con 0x7fd4b415e610 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac00c940 con 0x7fd4b415e610 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac00cd30 con 0x7fd4b415e610 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b415e100 con 0x7fd4b415e610 2024-03-21T08:41:01.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b414f700 con 0x7fd4b415e610 2024-03-21T08:41:01.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b404f410 con 0x7fd4b415e610 2024-03-21T08:41:01.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.703+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd4ac01a070 con 0x7fd4b415e610 2024-03-21T08:41:01.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.707+0000 7fd4abe00640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 0x7fd490069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:01.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.707+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4ac085bf0 con 0x7fd4b415e610 2024-03-21T08:41:01.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.707+0000 7fd4b3400640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 0x7fd490069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:01.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.707+0000 7fd4b3400640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 0x7fd490069fe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd4a40012f0 tx=0x7fd4a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.711+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd4ac056260 con 0x7fd4b415e610 2024-03-21T08:41:01.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.903+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4b4072c10 con 0x7fd4b415e610 2024-03-21T08:41:01.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.903+0000 7fd4abe00640 1 -- 172.21.15.28:0/1815941899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fd4ac05a300 con 0x7fd4b415e610 2024-03-21T08:41:01.906 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:01.906 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-21T08:41:01.907 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 msgr2=0x7fd490069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 0x7fd490069fe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd4a40012f0 tx=0x7fd4a4009040 comp rx=0 tx=0).stop 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 msgr2=0x7fd4b415d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e880 tx=0x7fd4ac00ed40 comp rx=0 tx=0).stop 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 shutdown_connections 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4b415e610 0x7fd4b415d4f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd490067b20 0x7fd490069fe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4b40ce490 0x7fd4b406e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 --2- 172.21.15.28:0/1815941899 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4b40b4380 0x7fd4b414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:01.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 >> 172.21.15.28:0/1815941899 conn(0x7fd4b40cc230 msgr2=0x7fd4b4163b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:01.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 shutdown_connections 2024-03-21T08:41:01.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:01.907+0000 7fd4ba000640 1 -- 172.21.15.28:0/1815941899 wait complete. 2024-03-21T08:41:03.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 Processor -- start 2024-03-21T08:41:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 -- start start 2024-03-21T08:41:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61c80ce490 0x7f61c815e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 0x7f61c80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c815bf00 con 0x7f61c815ba70 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c815ebf0 con 0x7f61c80ce490 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61cd000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c815eed0 con 0x7f61c80b4380 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61c80ce490 0x7f61c815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61c80ce490 0x7f61c815e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35354/0 (socket says 172.21.15.28:35354) 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61bd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f61c815ba70 0x7f61c80ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36184/0 (socket says 172.21.15.28:36184) 2024-03-21T08:41:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 -- 172.21.15.28:0/3213364087 learned_addr learned my addr 172.21.15.28:0/3213364087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c6800640 1 --2- 172.21.15.28:0/3213364087 >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 0x7f61c80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 -- 172.21.15.28:0/3213364087 >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 msgr2=0x7f61c80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 --2- 172.21.15.28:0/3213364087 >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 0x7f61c80b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 -- 172.21.15.28:0/3213364087 >> v1:172.21.15.28:6789/0 conn(0x7f61c815ba70 legacy=0x7f61c80ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c5e00640 1 -- 172.21.15.28:0/3213364087 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c815f1b0 con 0x7f61c80ce490 2024-03-21T08:41:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.079+0000 7f61c6800640 1 --2- 172.21.15.28:0/3213364087 >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 0x7f61c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:04.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61c5e00640 1 --2- 172.21.15.28:0/3213364087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80ce490 0x7f61c815e610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cf780 tx=0x7f61b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c23dc4f7c3a06 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61c5400640 1 -- 172.21.15.28:0/3213364087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80b2070 con 0x7f61c80ce490 2024-03-21T08:41:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61c5400640 1 -- 172.21.15.28:0/3213364087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b80ad680 con 0x7f61c80ce490 2024-03-21T08:41:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61c5400640 1 -- 172.21.15.28:0/3213364087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80ada50 con 0x7f61c80ce490 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 -- 172.21.15.28:0/3213364087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80ce490 msgr2=0x7f61c815e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 --2- 172.21.15.28:0/3213364087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80ce490 0x7f61c815e610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cf780 tx=0x7f61b801c490 comp rx=0 tx=0).stop 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 -- 172.21.15.28:0/3213364087 shutdown_connections 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 --2- 172.21.15.28:0/3213364087 >> v2:172.21.15.151:3300/0 conn(0x7f61c80b4380 0x7f61c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 --2- 172.21.15.28:0/3213364087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80ce490 0x7f61c815e610 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 -- 172.21.15.28:0/3213364087 >> 172.21.15.28:0/3213364087 conn(0x7f61c80cc230 msgr2=0x7f61c80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 -- 172.21.15.28:0/3213364087 shutdown_connections 2024-03-21T08:41:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.083+0000 7f61cd000640 1 -- 172.21.15.28:0/3213364087 wait complete. 2024-03-21T08:41:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 Processor -- start 2024-03-21T08:41:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- start start 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61c80ce490 0x7f61c814f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61c815ba70 0x7f61c8071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80cd500 con 0x7f61c80b4380 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80cd380 con 0x7f61c80ce490 2024-03-21T08:41:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c80cd680 con 0x7f61c815ba70 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35364/0 (socket says 172.21.15.28:35364) 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 -- 172.21.15.28:0/3090140560 learned_addr learned my addr 172.21.15.28:0/3090140560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61bd400640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61c80ce490 0x7f61c814f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c6800640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61c815ba70 0x7f61c8071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 -- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61c815ba70 msgr2=0x7f61c8071d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61c815ba70 0x7f61c8071d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 -- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61c80ce490 msgr2=0x7f61c814f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61c80ce490 0x7f61c814f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 -- 172.21.15.28:0/3090140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c81fb240 con 0x7f61c80b4380 2024-03-21T08:41:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61c5e00640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61b80190a0 tx=0x7f61b801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80b2040 con 0x7f61c80b4380 2024-03-21T08:41:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b80adb20 con 0x7f61c80b4380 2024-03-21T08:41:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b80b8450 con 0x7f61c80b4380 2024-03-21T08:41:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c8148290 con 0x7f61c80b4380 2024-03-21T08:41:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.087+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c8148b30 con 0x7f61c80b4380 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c804f410 con 0x7f61c80b4380 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f61b80bf030 con 0x7f61c80b4380 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61bea00640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 0x7f61a4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61b811f400 con 0x7f61c80b4380 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61bd400640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 0x7f61a4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.091+0000 7f61bd400640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 0x7f61a4069fe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61c814e300 tx=0x7f61b0001620 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.095+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f61b80c6060 con 0x7f61c80b4380 2024-03-21T08:41:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61c8151290 con 0x7f61c80b4380 2024-03-21T08:41:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61bea00640 1 -- 172.21.15.28:0/3090140560 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f61b8016110 con 0x7f61c80b4380 2024-03-21T08:41:04.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:04.297 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-21T08:41:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-03-21T08:41:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 msgr2=0x7f61a4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 0x7f61a4069fe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61c814e300 tx=0x7f61b0001620 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 msgr2=0x7f61c8143540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61b80190a0 tx=0x7f61b801acd0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 shutdown_connections 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f61a4067b20 0x7f61a4069fe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61c815ba70 0x7f61c8071d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61c80ce490 0x7f61c814f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 --2- 172.21.15.28:0/3090140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61c80b4380 0x7f61c8143540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 >> 172.21.15.28:0/3090140560 conn(0x7f61c80cc230 msgr2=0x7f61c814d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 shutdown_connections 2024-03-21T08:41:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.295+0000 7f61cd000640 1 -- 172.21.15.28:0/3090140560 wait complete. 2024-03-21T08:41:04.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.455+0000 7fca83c00640 1 Processor -- start 2024-03-21T08:41:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.455+0000 7fca83c00640 1 -- start start 2024-03-21T08:41:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca83c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c099b80 0x7fca7c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca83c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 0x7fca7c099f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca83c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c09a4a0 con 0x7fca7c0b0b40 2024-03-21T08:41:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca83c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c06d890 con 0x7fca7c099b80 2024-03-21T08:41:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca83c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cfac0 con 0x7fca7c0d0880 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c099b80 0x7fca7c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c099b80 0x7fca7c15f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35366/0 (socket says 172.21.15.28:35366) 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 -- 172.21.15.28:0/3282508780 learned_addr learned my addr 172.21.15.28:0/3282508780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca73e00640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 0x7fca7c099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 -- 172.21.15.28:0/3282508780 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 msgr2=0x7fca7c099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 0x7fca7c099f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 -- 172.21.15.28:0/3282508780 >> v1:172.21.15.151:6789/0 conn(0x7fca7c0b0b40 legacy=0x7fca7c161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca81200640 1 -- 172.21.15.28:0/3282508780 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0cfda0 con 0x7fca7c099b80 2024-03-21T08:41:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.459+0000 7fca73e00640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 0x7fca7c099f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca81200640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c099b80 0x7fca7c15f220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009870 tx=0x7fca6c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f31305c2ff9bf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca7be00640 1 -- 172.21.15.28:0/3282508780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0a7070 con 0x7fca7c099b80 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca7be00640 1 -- 172.21.15.28:0/3282508780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c0b0070 con 0x7fca7c099b80 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca7be00640 1 -- 172.21.15.28:0/3282508780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0ab890 con 0x7fca7c099b80 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- 172.21.15.28:0/3282508780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c099b80 msgr2=0x7fca7c15f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c099b80 0x7fca7c15f220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009870 tx=0x7fca6c01c530 comp rx=0 tx=0).stop 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- 172.21.15.28:0/3282508780 shutdown_connections 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0d0880 0x7fca7c099f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 --2- 172.21.15.28:0/3282508780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c099b80 0x7fca7c15f220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- 172.21.15.28:0/3282508780 >> 172.21.15.28:0/3282508780 conn(0x7fca7c0b3510 msgr2=0x7fca7c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- 172.21.15.28:0/3282508780 shutdown_connections 2024-03-21T08:41:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- 172.21.15.28:0/3282508780 wait complete. 2024-03-21T08:41:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 Processor -- start 2024-03-21T08:41:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.463+0000 7fca83c00640 1 -- start start 2024-03-21T08:41:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 0x7fca7c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0b0b40 0x7fca7c0b6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0b2cb0 con 0x7fca7c0d0880 2024-03-21T08:41:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0b2b30 con 0x7fca7c099b80 2024-03-21T08:41:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c16b050 con 0x7fca7c0b0b40 2024-03-21T08:41:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35376/0 (socket says 172.21.15.28:35376) 2024-03-21T08:41:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 -- 172.21.15.28:0/4087709868 learned_addr learned my addr 172.21.15.28:0/4087709868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca73e00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0b0b40 0x7fca7c0b6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81200640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 0x7fca7c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 -- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0b0b40 msgr2=0x7fca7c0b6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0b0b40 0x7fca7c0b6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 -- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 msgr2=0x7fca7c15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 0x7fca7c15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 -- 172.21.15.28:0/4087709868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0b7520 con 0x7fca7c0d0880 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81200640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 0x7fca7c15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca81c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca7400c770 tx=0x7fca7400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74003b80 con 0x7fca7c0d0880 2024-03-21T08:41:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca74003d20 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c0b5f40 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400aa90 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c0b6500 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c04f410 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.467+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fca7400ac30 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.471+0000 7fca79600640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 0x7fca58069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.471+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fca74085ab0 con 0x7fca7c0d0880 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.471+0000 7fca81200640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 0x7fca58069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.471+0000 7fca81200640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 0x7fca58069fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af000 tx=0x7fca6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.475+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fca74056810 con 0x7fca7c0d0880 2024-03-21T08:41:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca7c0b2080 con 0x7fca7c0d0880 2024-03-21T08:41:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca79600640 1 -- 172.21.15.28:0/4087709868 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7fca74056110 con 0x7fca7c0d0880 2024-03-21T08:41:04.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:04.640 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:40:55.899749+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 msgr2=0x7fca58069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 0x7fca58069fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0af000 tx=0x7fca6c008040 comp rx=0 tx=0).stop 2024-03-21T08:41:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 msgr2=0x7fca7c0aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca7400c770 tx=0x7fca7400cc30 comp rx=0 tx=0).stop 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 shutdown_connections 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0d0880 0x7fca7c0aeaf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fca58067b20 0x7fca58069fe0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fca7c0b0b40 0x7fca7c0b6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 --2- 172.21.15.28:0/4087709868 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fca7c099b80 0x7fca7c15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 >> 172.21.15.28:0/4087709868 conn(0x7fca7c0b3510 msgr2=0x7fca7c070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 shutdown_connections 2024-03-21T08:41:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.639+0000 7fca83c00640 1 -- 172.21.15.28:0/4087709868 wait complete. 2024-03-21T08:41:04.662 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:41:04.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.803+0000 7efd67800640 1 Processor -- start 2024-03-21T08:41:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.803+0000 7efd67800640 1 -- start start 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd67800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd67800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efd6006eb40 0x7efd6006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd67800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd600d1790 con 0x7efd60162430 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd67800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600cd7d0 con 0x7efd6006eb40 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd67800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd600cdab0 con 0x7efd600d13b0 2024-03-21T08:41:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd5fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd60162430 0x7efd600af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36200/0 (socket says 172.21.15.28:36200) 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd5fe00640 1 -- 172.21.15.28:0/2302527713 learned_addr learned my addr 172.21.15.28:0/2302527713 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd64e00640 1 --2- 172.21.15.28:0/2302527713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd65800640 1 --2- 172.21.15.28:0/2302527713 >> v2:172.21.15.151:3300/0 conn(0x7efd6006eb40 0x7efd6006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd5f400640 1 -- 172.21.15.28:0/2302527713 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879073436 0 0) 0x7efd600d1790 con 0x7efd60162430 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd5f400640 1 -- 172.21.15.28:0/2302527713 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd48003530 con 0x7efd60162430 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd64e00640 1 -- 172.21.15.28:0/2302527713 >> v2:172.21.15.151:3300/0 conn(0x7efd6006eb40 msgr2=0x7efd6006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd64e00640 1 --2- 172.21.15.28:0/2302527713 >> v2:172.21.15.151:3300/0 conn(0x7efd6006eb40 0x7efd6006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd64e00640 1 -- 172.21.15.28:0/2302527713 >> v1:172.21.15.28:6789/0 conn(0x7efd60162430 legacy=0x7efd600af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.807+0000 7efd64e00640 1 -- 172.21.15.28:0/2302527713 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600cdd90 con 0x7efd600d13b0 2024-03-21T08:41:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd64e00640 1 --2- 172.21.15.28:0/2302527713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60161ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd540097c0 tx=0x7efd5401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f529e4e14fac04 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd5f400640 1 -- 172.21.15.28:0/2302527713 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540b1070 con 0x7efd600d13b0 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd5f400640 1 -- 172.21.15.28:0/2302527713 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd540ac410 con 0x7efd600d13b0 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd5f400640 1 -- 172.21.15.28:0/2302527713 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540ac7c0 con 0x7efd600d13b0 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- 172.21.15.28:0/2302527713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 msgr2=0x7efd60161ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- 172.21.15.28:0/2302527713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60161ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd540097c0 tx=0x7efd5401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- 172.21.15.28:0/2302527713 shutdown_connections 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- 172.21.15.28:0/2302527713 >> v2:172.21.15.151:3300/0 conn(0x7efd6006eb40 0x7efd6006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- 172.21.15.28:0/2302527713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60161ef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- 172.21.15.28:0/2302527713 >> 172.21.15.28:0/2302527713 conn(0x7efd60099a30 msgr2=0x7efd60099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- 172.21.15.28:0/2302527713 shutdown_connections 2024-03-21T08:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- 172.21.15.28:0/2302527713 wait complete. 2024-03-21T08:41:04.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 Processor -- start 2024-03-21T08:41:04.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- start start 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd60162430 0x7efd601528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6015e5e0 con 0x7efd6006eb40 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b0230 con 0x7efd60162430 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd67800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6015e760 con 0x7efd600d13b0 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35386/0 (socket says 172.21.15.28:35386) 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.811+0000 7efd64e00640 1 -- 172.21.15.28:0/4151402329 learned_addr learned my addr 172.21.15.28:0/4151402329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5fe00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd65800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd60162430 0x7efd601528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 -- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 msgr2=0x7efd60152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 -- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd60162430 msgr2=0x7efd601528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd60162430 0x7efd601528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 -- 172.21.15.28:0/4151402329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600aeeb0 con 0x7efd6006eb40 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd64e00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd5401a0d0 tx=0x7efd54019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540b9020 con 0x7efd6006eb40 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd540accc0 con 0x7efd6006eb40 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd600af190 con 0x7efd6006eb40 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540b1040 con 0x7efd6006eb40 2024-03-21T08:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd600af750 con 0x7efd6006eb40 2024-03-21T08:41:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6009ea60 con 0x7efd6006eb40 2024-03-21T08:41:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7efd540bc020 con 0x7efd6006eb40 2024-03-21T08:41:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 0x7efd40069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.815+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efd5411f460 con 0x7efd6006eb40 2024-03-21T08:41:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.819+0000 7efd5fe00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 0x7efd40069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.819+0000 7efd5fe00640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 0x7efd40069fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd6015f120 tx=0x7efd4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.819+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efd540c3030 con 0x7efd6006eb40 2024-03-21T08:41:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd600cd0e0 con 0x7efd6006eb40 2024-03-21T08:41:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd5cc00640 1 -- 172.21.15.28:0/4151402329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd54016090 con 0x7efd6006eb40 2024-03-21T08:41:04.988 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 msgr2=0x7efd40069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 0x7efd40069fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd6015f120 tx=0x7efd4c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 msgr2=0x7efd6014bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd5401a0d0 tx=0x7efd54019240 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 shutdown_connections 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efd60162430 0x7efd601528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efd40067b20 0x7efd40069fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efd600d13b0 0x7efd60152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 --2- 172.21.15.28:0/4151402329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6006eb40 0x7efd6014bce0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 >> 172.21.15.28:0/4151402329 conn(0x7efd60099a30 msgr2=0x7efd600b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 shutdown_connections 2024-03-21T08:41:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:04.987+0000 7efd67800640 1 -- 172.21.15.28:0/4151402329 wait complete. 2024-03-21T08:41:05.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-21T08:41:05.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 Processor -- start 2024-03-21T08:41:05.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 -- start start 2024-03-21T08:41:05.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb980b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980cd1d0 con 0x7fbb98099a30 2024-03-21T08:41:05.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980cd4b0 con 0x7fbb980d0480 2024-03-21T08:41:05.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980ccf90 con 0x7fbb9815f220 2024-03-21T08:41:05.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbb98099a30 0x7fbb980c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36214/0 (socket says 172.21.15.28:36214) 2024-03-21T08:41:05.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9d000640 1 -- 172.21.15.28:0/2645581095 learned_addr learned my addr 172.21.15.28:0/2645581095 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:05.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9da00640 1 --2- 172.21.15.28:0/2645581095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb980b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260547415 0 0) 0x7fbb980cd1d0 con 0x7fbb98099a30 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb80003530 con 0x7fbb98099a30 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713070759 0 0) 0x7fbb980cd4b0 con 0x7fbb980d0480 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb980cd1d0 con 0x7fbb980d0480 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9da00640 1 -- 172.21.15.28:0/2645581095 >> v1:172.21.15.151:6789/0 conn(0x7fbb980d0480 legacy=0x7fbb980c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9da00640 1 -- 172.21.15.28:0/2645581095 >> v1:172.21.15.28:6789/0 conn(0x7fbb98099a30 legacy=0x7fbb980c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.151+0000 7fbb9da00640 1 -- 172.21.15.28:0/2645581095 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb980cd790 con 0x7fbb9815f220 2024-03-21T08:41:05.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9da00640 1 --2- 172.21.15.28:0/2645581095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb980b2770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097f0 tx=0x7fbb8c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114f3dd08d8d7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0b2070 con 0x7fbb9815f220 2024-03-21T08:41:05.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c0a9070 con 0x7fbb9815f220 2024-03-21T08:41:05.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb97e00640 1 -- 172.21.15.28:0/2645581095 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0ad880 con 0x7fbb9815f220 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 -- 172.21.15.28:0/2645581095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 msgr2=0x7fbb980b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/2645581095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb980b2770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0097f0 tx=0x7fbb8c01c4e0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 -- 172.21.15.28:0/2645581095 shutdown_connections 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/2645581095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb980b2770 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 -- 172.21.15.28:0/2645581095 >> 172.21.15.28:0/2645581095 conn(0x7fbb980b3390 msgr2=0x7fbb980b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:05.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 -- 172.21.15.28:0/2645581095 shutdown_connections 2024-03-21T08:41:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 -- 172.21.15.28:0/2645581095 wait complete. 2024-03-21T08:41:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.155+0000 7fbb9ee00640 1 Processor -- start 2024-03-21T08:41:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- start start 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb98099a30 0x7fbb981580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb981567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9809a3c0 con 0x7fbb980d0480 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9809a240 con 0x7fbb98099a30 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980b2db0 con 0x7fbb9815f220 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35402/0 (socket says 172.21.15.28:35402) 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 -- 172.21.15.28:0/219235103 learned_addr learned my addr 172.21.15.28:0/219235103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9da00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb98099a30 0x7fbb981580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9e400640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb981567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 -- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 msgr2=0x7fbb981567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb981567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 -- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb98099a30 msgr2=0x7fbb981580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb98099a30 0x7fbb981580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 -- 172.21.15.28:0/219235103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98156ce0 con 0x7fbb980d0480 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9e400640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb981567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9d000640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400c4d0 tx=0x7fbb8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84017070 con 0x7fbb980d0480 2024-03-21T08:41:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84005cf0 con 0x7fbb980d0480 2024-03-21T08:41:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84007290 con 0x7fbb980d0480 2024-03-21T08:41:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb98157640 con 0x7fbb980d0480 2024-03-21T08:41:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb980c92e0 con 0x7fbb980d0480 2024-03-21T08:41:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.159+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb98075270 con 0x7fbb980d0480 2024-03-21T08:41:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.163+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbb8401a070 con 0x7fbb980d0480 2024-03-21T08:41:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.163+0000 7fbb95600640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 0x7fbb7806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.163+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb84084ee0 con 0x7fbb980d0480 2024-03-21T08:41:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.163+0000 7fbb9da00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 0x7fbb7806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.163+0000 7fbb9da00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 0x7fbb7806a010 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0af000 tx=0x7fbb8c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.167+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbb84055cd0 con 0x7fbb980d0480 2024-03-21T08:41:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbb980c5dc0 con 0x7fbb980d0480 2024-03-21T08:41:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb95600640 1 -- 172.21.15.28:0/219235103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbb840555d0 con 0x7fbb980d0480 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-03-21T08:41:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:41:05.368 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 msgr2=0x7fbb7806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 0x7fbb7806a010 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0af000 tx=0x7fbb8c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 msgr2=0x7fbb980c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400c4d0 tx=0x7fbb8400c990 comp rx=0 tx=0).stop 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 shutdown_connections 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbb78067b50 0x7fbb7806a010 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbb9815f220 0x7fbb981567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb980d0480 0x7fbb980c8970 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 --2- 172.21.15.28:0/219235103 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbb98099a30 0x7fbb981580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 >> 172.21.15.28:0/219235103 conn(0x7fbb980b3390 msgr2=0x7fbb980cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 shutdown_connections 2024-03-21T08:41:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.331+0000 7fbb9ee00640 1 -- 172.21.15.28:0/219235103 wait complete. 2024-03-21T08:41:05.370 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-21T08:41:05.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-21T08:41:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 Processor -- start 2024-03-21T08:41:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 -- start start 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48f006e180 0x7f48f006e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f00cfbc0 con 0x7f48f01585d0 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f015b1a0 con 0x7f48f00ce0b0 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48f5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f006eb20 con 0x7f48f006e180 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48f006e180 0x7f48f006e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48f006e180 0x7f48f006e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35412/0 (socket says 172.21.15.28:35412) 2024-03-21T08:41:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 -- 172.21.15.28:0/1430919598 learned_addr learned my addr 172.21.15.28:0/1430919598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 -- 172.21.15.28:0/1430919598 >> v1:172.21.15.151:6789/0 conn(0x7f48f00ce0b0 legacy=0x7f48f00ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 -- 172.21.15.28:0/1430919598 >> v1:172.21.15.28:6789/0 conn(0x7f48f01585d0 legacy=0x7f48f00cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.515+0000 7f48e7e00640 1 -- 172.21.15.28:0/1430919598 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f015b480 con 0x7f48f006e180 2024-03-21T08:41:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48e7e00640 1 --2- 172.21.15.28:0/1430919598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f006e180 0x7f48f006e5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009870 tx=0x7f48d801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722c1e521446770f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48ef400640 1 -- 172.21.15.28:0/1430919598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d80a95f0 con 0x7f48f006e180 2024-03-21T08:41:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48ef400640 1 -- 172.21.15.28:0/1430919598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d80a9790 con 0x7f48f006e180 2024-03-21T08:41:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48ef400640 1 -- 172.21.15.28:0/1430919598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d80a9b60 con 0x7f48f006e180 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- 172.21.15.28:0/1430919598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f006e180 msgr2=0x7f48f006e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 --2- 172.21.15.28:0/1430919598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f006e180 0x7f48f006e5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009870 tx=0x7f48d801c530 comp rx=0 tx=0).stop 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- 172.21.15.28:0/1430919598 shutdown_connections 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 --2- 172.21.15.28:0/1430919598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f006e180 0x7f48f006e5e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- 172.21.15.28:0/1430919598 >> 172.21.15.28:0/1430919598 conn(0x7f48f00b1db0 msgr2=0x7f48f00b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- 172.21.15.28:0/1430919598 shutdown_connections 2024-03-21T08:41:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- 172.21.15.28:0/1430919598 wait complete. 2024-03-21T08:41:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 Processor -- start 2024-03-21T08:41:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.519+0000 7f48f5a00640 1 -- start start 2024-03-21T08:41:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f48f006e180 0x7f48f014fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48f00ce0b0 0x7f48f0159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f015bd20 con 0x7f48f01585d0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f006b170 con 0x7f48f00ce0b0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0072d00 con 0x7f48f006e180 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35424/0 (socket says 172.21.15.28:35424) 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 -- 172.21.15.28:0/2943699252 learned_addr learned my addr 172.21.15.28:0/2943699252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48e7e00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48f00ce0b0 0x7f48f0159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48efe00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f48f006e180 0x7f48f014fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 -- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f48f006e180 msgr2=0x7f48f014fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f48f006e180 0x7f48f014fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 -- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48f00ce0b0 msgr2=0x7f48f0159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48f00ce0b0 0x7f48f0159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 -- 172.21.15.28:0/2943699252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f0156120 con 0x7f48f01585d0 2024-03-21T08:41:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5000640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48e800c770 tx=0x7f48e800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8003b80 con 0x7f48f01585d0 2024-03-21T08:41:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e8003d20 con 0x7f48f01585d0 2024-03-21T08:41:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800aa90 con 0x7f48f01585d0 2024-03-21T08:41:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f0156400 con 0x7f48f01585d0 2024-03-21T08:41:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f0150310 con 0x7f48f01585d0 2024-03-21T08:41:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.523+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f004f410 con 0x7f48f01585d0 2024-03-21T08:41:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.527+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f48e800ac30 con 0x7f48f01585d0 2024-03-21T08:41:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.527+0000 7f48ecc00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 0x7f48cc06a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.527+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f48e8085950 con 0x7f48f01585d0 2024-03-21T08:41:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.527+0000 7f48efe00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 0x7f48cc06a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:05.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.527+0000 7f48efe00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 0x7f48cc06a030 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48e0007fd0 tx=0x7f48e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:05.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.531+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f48e80560d0 con 0x7f48f01585d0 2024-03-21T08:41:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.699+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f48f0155370 con 0x7f48f01585d0 2024-03-21T08:41:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48ecc00640 1 -- 172.21.15.28:0/2943699252 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f48e805a170 con 0x7f48f01585d0 2024-03-21T08:41:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-03-21T08:41:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 msgr2=0x7f48cc06a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 0x7f48cc06a030 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48e0007fd0 tx=0x7f48e0009040 comp rx=0 tx=0).stop 2024-03-21T08:41:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 msgr2=0x7f48f0152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48e800c770 tx=0x7f48e800cc30 comp rx=0 tx=0).stop 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 shutdown_connections 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f01585d0 0x7f48f0152d30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f48f00ce0b0 0x7f48f0159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f48cc067b70 0x7f48cc06a030 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 --2- 172.21.15.28:0/2943699252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f48f006e180 0x7f48f014fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 >> 172.21.15.28:0/2943699252 conn(0x7f48f00b1db0 msgr2=0x7f48f0151220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 shutdown_connections 2024-03-21T08:41:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:05.975+0000 7f48f5a00640 1 -- 172.21.15.28:0/2943699252 wait complete. 2024-03-21T08:41:06.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-21T08:41:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.147+0000 7f8287000640 1 Processor -- start 2024-03-21T08:41:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 -- start start 2024-03-21T08:41:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f828015e1e0 0x7f82800cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828006cbe0 0x7f8280161c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800d05f0 con 0x7f828006cbe0 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828006d070 con 0x7f828015e1e0 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8287000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82800b3010 con 0x7f82800d0190 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828006cbe0 0x7f8280161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828006cbe0 0x7f8280161c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35434/0 (socket says 172.21.15.28:35434) 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 -- 172.21.15.28:0/3395555437 learned_addr learned my addr 172.21.15.28:0/3395555437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f827fe00640 1 --2- 172.21.15.28:0/3395555437 >> v2:172.21.15.151:3300/0 conn(0x7f828015e1e0 0x7f82800cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f827f400640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 -- 172.21.15.28:0/3395555437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 msgr2=0x7f828006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828006c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 -- 172.21.15.28:0/3395555437 >> v2:172.21.15.151:3300/0 conn(0x7f828015e1e0 msgr2=0x7f82800cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 --2- 172.21.15.28:0/3395555437 >> v2:172.21.15.151:3300/0 conn(0x7f828015e1e0 0x7f82800cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.151+0000 7f8285000640 1 -- 172.21.15.28:0/3395555437 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828006d310 con 0x7f828006cbe0 2024-03-21T08:41:06.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8285000640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f8280161c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8280068b20 tx=0x7f827801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a000802901bcb82a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f827ea00640 1 -- 172.21.15.28:0/3395555437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82780a64b0 con 0x7f828006cbe0 2024-03-21T08:41:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f827ea00640 1 -- 172.21.15.28:0/3395555437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82780a6650 con 0x7f828006cbe0 2024-03-21T08:41:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f827ea00640 1 -- 172.21.15.28:0/3395555437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82780a6a20 con 0x7f828006cbe0 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- 172.21.15.28:0/3395555437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 msgr2=0x7f8280161c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f8280161c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8280068b20 tx=0x7f827801c730 comp rx=0 tx=0).stop 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- 172.21.15.28:0/3395555437 shutdown_connections 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f8280161c00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 --2- 172.21.15.28:0/3395555437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828006c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 --2- 172.21.15.28:0/3395555437 >> v2:172.21.15.151:3300/0 conn(0x7f828015e1e0 0x7f82800cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- 172.21.15.28:0/3395555437 >> 172.21.15.28:0/3395555437 conn(0x7f82800d1ab0 msgr2=0x7f82800d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- 172.21.15.28:0/3395555437 shutdown_connections 2024-03-21T08:41:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- 172.21.15.28:0/3395555437 wait complete. 2024-03-21T08:41:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 Processor -- start 2024-03-21T08:41:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.155+0000 7f8287000640 1 -- start start 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828013fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f828015e1e0 0x7f8280140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800cc660 con 0x7f828006cbe0 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800cc4e0 con 0x7f828015e1e0 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82800cc7e0 con 0x7f82800d0190 2024-03-21T08:41:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35436/0 (socket says 172.21.15.28:35436) 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 -- 172.21.15.28:0/3562358511 learned_addr learned my addr 172.21.15.28:0/3562358511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827f400640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828013fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8285000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f828015e1e0 0x7f8280140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 -- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 msgr2=0x7f828013fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828013fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 -- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f828015e1e0 msgr2=0x7f8280140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f828015e1e0 0x7f8280140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 -- 172.21.15.28:0/3562358511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82801532b0 con 0x7f828006cbe0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f827fe00640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f827000e870 tx=0x7f827000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270017070 con 0x7f828006cbe0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827000c940 con 0x7f828006cbe0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827000ccf0 con 0x7f828006cbe0 2024-03-21T08:41:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82801406c0 con 0x7f828006cbe0 2024-03-21T08:41:06.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82801520d0 con 0x7f828006cbe0 2024-03-21T08:41:06.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.159+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828004f4f0 con 0x7f828006cbe0 2024-03-21T08:41:06.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.163+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f827001a070 con 0x7f828006cbe0 2024-03-21T08:41:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.163+0000 7f8277e00640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 0x7f8260069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.163+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8270085b50 con 0x7f828006cbe0 2024-03-21T08:41:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.163+0000 7f827f400640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 0x7f8260069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.163+0000 7f827f400640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 0x7f8260069fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82800b7b00 tx=0x7f826c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:06.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.167+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8270056050 con 0x7f828006cbe0 2024-03-21T08:41:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.331+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f82800d2830 con 0x7f828006cbe0 2024-03-21T08:41:06.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.979+0000 7f8277e00640 1 -- 172.21.15.28:0/3562358511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7f827005a0f0 con 0x7f828006cbe0 2024-03-21T08:41:06.983 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 msgr2=0x7f8260069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 0x7f8260069fe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82800b7b00 tx=0x7f826c008040 comp rx=0 tx=0).stop 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 msgr2=0x7f82801525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f827000e870 tx=0x7f827000ec70 comp rx=0 tx=0).stop 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 shutdown_connections 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f828015e1e0 0x7f8280140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8260067b20 0x7f8260069fe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82800d0190 0x7f828013fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 --2- 172.21.15.28:0/3562358511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828006cbe0 0x7f82801525f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.983+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 >> 172.21.15.28:0/3562358511 conn(0x7f82800d1ab0 msgr2=0x7f82800ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.987+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 shutdown_connections 2024-03-21T08:41:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:06.987+0000 7f8287000640 1 -- 172.21.15.28:0/3562358511 wait complete. 2024-03-21T08:41:07.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-21T08:41:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 Processor -- start 2024-03-21T08:41:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 -- start start 2024-03-21T08:41:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe65c0d0be0 0x7fe65c0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0d1380 con 0x7fe65c06cc90 2024-03-21T08:41:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0d1660 con 0x7fe65c0d0750 2024-03-21T08:41:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe660c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c0cc220 con 0x7fe65c0d0be0 2024-03-21T08:41:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe659e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe65c06cc90 0x7fe65c06d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36242/0 (socket says 172.21.15.28:36242) 2024-03-21T08:41:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe659e00640 1 -- 172.21.15.28:0/4182683032 learned_addr learned my addr 172.21.15.28:0/4182683032 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe65a800640 1 --2- 172.21.15.28:0/4182683032 >> v2:172.21.15.28:3300/0 conn(0x7fe65c0d0be0 0x7fe65c0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe658a00640 1 -- 172.21.15.28:0/4182683032 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426598412 0 0) 0x7fe65c0d1380 con 0x7fe65c06cc90 2024-03-21T08:41:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe658a00640 1 -- 172.21.15.28:0/4182683032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640003530 con 0x7fe65c06cc90 2024-03-21T08:41:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe65a800640 1 -- 172.21.15.28:0/4182683032 >> v1:172.21.15.151:6789/0 conn(0x7fe65c0d0750 legacy=0x7fe65c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe65a800640 1 -- 172.21.15.28:0/4182683032 >> v1:172.21.15.28:6789/0 conn(0x7fe65c06cc90 legacy=0x7fe65c06d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.179+0000 7fe65a800640 1 -- 172.21.15.28:0/4182683032 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0cc500 con 0x7fe65c0d0be0 2024-03-21T08:41:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe65a800640 1 --2- 172.21.15.28:0/4182683032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0be0 0x7fe65c0cd130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe654009dc0 tx=0x7fe65401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b895b8cffa530e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe658a00640 1 -- 172.21.15.28:0/4182683032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540b2070 con 0x7fe65c0d0be0 2024-03-21T08:41:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe658a00640 1 -- 172.21.15.28:0/4182683032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6540a5890 con 0x7fe65c0d0be0 2024-03-21T08:41:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe658a00640 1 -- 172.21.15.28:0/4182683032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540a5c40 con 0x7fe65c0d0be0 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- 172.21.15.28:0/4182683032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0be0 msgr2=0x7fe65c0cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 --2- 172.21.15.28:0/4182683032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0be0 0x7fe65c0cd130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe654009dc0 tx=0x7fe65401c8c0 comp rx=0 tx=0).stop 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- 172.21.15.28:0/4182683032 shutdown_connections 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 --2- 172.21.15.28:0/4182683032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0be0 0x7fe65c0cd130 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- 172.21.15.28:0/4182683032 >> 172.21.15.28:0/4182683032 conn(0x7fe65c0b3540 msgr2=0x7fe65c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- 172.21.15.28:0/4182683032 shutdown_connections 2024-03-21T08:41:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- 172.21.15.28:0/4182683032 wait complete. 2024-03-21T08:41:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 Processor -- start 2024-03-21T08:41:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.183+0000 7fe660c00640 1 -- start start 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe65c06cc90 0x7fe65c0c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe65c0d0be0 0x7fe65c0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c16bdb0 con 0x7fe65c0d0750 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c16bc30 con 0x7fe65c06cc90 2024-03-21T08:41:07.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c16bf30 con 0x7fe65c0d0be0 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35460/0 (socket says 172.21.15.28:35460) 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 -- 172.21.15.28:0/2076175624 learned_addr learned my addr 172.21.15.28:0/2076175624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe65a800640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe65c0d0be0 0x7fe65c0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659e00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe65c06cc90 0x7fe65c0c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 -- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe65c0d0be0 msgr2=0x7fe65c0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe65c0d0be0 0x7fe65c0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 -- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe65c06cc90 msgr2=0x7fe65c0c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe65c06cc90 0x7fe65c0c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 -- 172.21.15.28:0/2076175624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0c9060 con 0x7fe65c0d0750 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe659400640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe64800c370 tx=0x7fe64800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480052d0 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe648005470 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c0c92e0 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648007260 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c0c9b80 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.187+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c09f9c0 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.191+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe648007400 con 0x7fe65c0d0750 2024-03-21T08:41:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.191+0000 7fe652000640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 0x7fe63c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.191+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe648085730 con 0x7fe65c0d0750 2024-03-21T08:41:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.191+0000 7fe659e00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 0x7fe63c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.191+0000 7fe659e00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 0x7fe63c069fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0011f0 tx=0x7fe64c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:07.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.195+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe65c0c9060 con 0x7fe65c0d0750 2024-03-21T08:41:07.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.391+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe65c15d960 con 0x7fe65c0d0750 2024-03-21T08:41:07.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:41:07.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:41:07.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.991+0000 7fe652000640 1 -- 172.21.15.28:0/2076175624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe648055bc0 con 0x7fe65c0d0750 2024-03-21T08:41:07.994 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-03-21T08:41:07.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 msgr2=0x7fe63c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 0x7fe63c069fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0011f0 tx=0x7fe64c007d40 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 msgr2=0x7fe65c0c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe64800c370 tx=0x7fe64800c830 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 shutdown_connections 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe63c067b20 0x7fe63c069fe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe65c0d0be0 0x7fe65c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe65c0d0750 0x7fe65c0c7e10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 --2- 172.21.15.28:0/2076175624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe65c06cc90 0x7fe65c0c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.995+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 >> 172.21.15.28:0/2076175624 conn(0x7fe65c0b3540 msgr2=0x7fe65c06d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.999+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 shutdown_connections 2024-03-21T08:41:08.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:07.999+0000 7fe660c00640 1 -- 172.21.15.28:0/2076175624 wait complete. 2024-03-21T08:41:08.033 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-21T08:41:08.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-21T08:41:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.175+0000 7fb07e200640 1 Processor -- start 2024-03-21T08:41:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb07e200640 1 -- start start 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb07e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb07809a330 0x7fb07806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb07e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0780d1600 con 0x7fb0780cfd50 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb07e200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0780cc180 con 0x7fb07815e050 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb07e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0780ccab0 con 0x7fb07809a330 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb076a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb07809a330 0x7fb07806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb077e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0780cfd50 0x7fb0780d01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36258/0 (socket says 172.21.15.28:36258) 2024-03-21T08:41:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb077e00640 1 -- 172.21.15.28:0/2100954005 learned_addr learned my addr 172.21.15.28:0/2100954005 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb076a00640 1 -- 172.21.15.28:0/2100954005 >> v1:172.21.15.151:6789/0 conn(0x7fb07815e050 legacy=0x7fb078099c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb076a00640 1 -- 172.21.15.28:0/2100954005 >> v1:172.21.15.28:6789/0 conn(0x7fb0780cfd50 legacy=0x7fb0780d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb076a00640 1 -- 172.21.15.28:0/2100954005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0780cc460 con 0x7fb07809a330 2024-03-21T08:41:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.179+0000 7fb076000640 1 -- 172.21.15.28:0/2100954005 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801782331 0 0) 0x7fb0780d1600 con 0x7fb0780cfd50 2024-03-21T08:41:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb076a00640 1 --2- 172.21.15.28:0/2100954005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07809a330 0x7fb07806d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097c0 tx=0x7fb06401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1892af462b9c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:08.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb076000640 1 -- 172.21.15.28:0/2100954005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640ab430 con 0x7fb07809a330 2024-03-21T08:41:08.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb076000640 1 -- 172.21.15.28:0/2100954005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0640b0070 con 0x7fb07809a330 2024-03-21T08:41:08.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb076000640 1 -- 172.21.15.28:0/2100954005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640ab7f0 con 0x7fb07809a330 2024-03-21T08:41:08.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- 172.21.15.28:0/2100954005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07809a330 msgr2=0x7fb07806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 --2- 172.21.15.28:0/2100954005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07809a330 0x7fb07806d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097c0 tx=0x7fb06401c410 comp rx=0 tx=0).stop 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- 172.21.15.28:0/2100954005 shutdown_connections 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 --2- 172.21.15.28:0/2100954005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07809a330 0x7fb07806d4f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- 172.21.15.28:0/2100954005 >> 172.21.15.28:0/2100954005 conn(0x7fb0780b34f0 msgr2=0x7fb0780b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- 172.21.15.28:0/2100954005 shutdown_connections 2024-03-21T08:41:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- 172.21.15.28:0/2100954005 wait complete. 2024-03-21T08:41:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 Processor -- start 2024-03-21T08:41:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.183+0000 7fb07e200640 1 -- start start 2024-03-21T08:41:08.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07809a330 0x7fb07814d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07815e050 0x7fb078158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780d1310 con 0x7fb0780cfd50 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07816bab0 con 0x7fb07815e050 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07814dfc0 con 0x7fb07809a330 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35484/0 (socket says 172.21.15.28:35484) 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 -- 172.21.15.28:0/673114037 learned_addr learned my addr 172.21.15.28:0/673114037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb077e00640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07815e050 0x7fb078158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb077400640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07809a330 0x7fb07814d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 -- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07809a330 msgr2=0x7fb07814d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07809a330 0x7fb07814d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 -- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07815e050 msgr2=0x7fb078158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07815e050 0x7fb078158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 -- 172.21.15.28:0/673114037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078154470 con 0x7fb0780cfd50 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb076a00640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0640ad040 tx=0x7fb06401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640bf610 con 0x7fb0780cfd50 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0640bf7d0 con 0x7fb0780cfd50 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640bfbc0 con 0x7fb0780cfd50 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb078154750 con 0x7fb0780cfd50 2024-03-21T08:41:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb078159ff0 con 0x7fb0780cfd50 2024-03-21T08:41:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb078159e00 con 0x7fb0780cfd50 2024-03-21T08:41:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.187+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb0640bd020 con 0x7fb0780cfd50 2024-03-21T08:41:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.191+0000 7fb06f400640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 0x7fb058069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.191+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0640a8020 con 0x7fb0780cfd50 2024-03-21T08:41:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.191+0000 7fb077400640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 0x7fb058069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:08.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.191+0000 7fb077400640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 0x7fb058069fe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb07815d220 tx=0x7fb068009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.195+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb064016090 con 0x7fb0780cfd50 2024-03-21T08:41:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.363+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fb078068110 con 0x7fb0780cfd50 2024-03-21T08:41:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb06f400640 1 -- 172.21.15.28:0/673114037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7fb0640f0b20 con 0x7fb0780cfd50 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 msgr2=0x7fb058069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 0x7fb058069fe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb07815d220 tx=0x7fb068009040 comp rx=0 tx=0).stop 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 msgr2=0x7fb078156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb0640ad040 tx=0x7fb06401ac50 comp rx=0 tx=0).stop 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 shutdown_connections 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb07815e050 0x7fb078158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0780cfd50 0x7fb078156820 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb058067b20 0x7fb058069fe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 --2- 172.21.15.28:0/673114037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb07809a330 0x7fb07814d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 >> 172.21.15.28:0/673114037 conn(0x7fb0780b34f0 msgr2=0x7fb0780714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 shutdown_connections 2024-03-21T08:41:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:08.987+0000 7fb07e200640 1 -- 172.21.15.28:0/673114037 wait complete. 2024-03-21T08:41:09.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-21T08:41:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 Processor -- start 2024-03-21T08:41:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 -- start start 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd74071a80 0x7fdd7406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdd740cde40 0x7fdd74071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd740723a0 con 0x7fdd7415cbb0 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7415c720 con 0x7fdd74071a80 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd79800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd740ce5b0 con 0x7fdd740cde40 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd74071a80 0x7fdd7406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd74071a80 0x7fdd7406f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35490/0 (socket says 172.21.15.28:35490) 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 -- 172.21.15.28:0/342373415 learned_addr learned my addr 172.21.15.28:0/342373415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd73400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd7415cbb0 0x7fdd740cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36266/0 (socket says 172.21.15.28:36266) 2024-03-21T08:41:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72000640 1 --2- 172.21.15.28:0/342373415 >> v2:172.21.15.151:3300/0 conn(0x7fdd740cde40 0x7fdd74071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 -- 172.21.15.28:0/342373415 >> v2:172.21.15.151:3300/0 conn(0x7fdd740cde40 msgr2=0x7fdd74071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 --2- 172.21.15.28:0/342373415 >> v2:172.21.15.151:3300/0 conn(0x7fdd740cde40 0x7fdd74071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 -- 172.21.15.28:0/342373415 >> v1:172.21.15.28:6789/0 conn(0x7fdd7415cbb0 legacy=0x7fdd740cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd72a00640 1 -- 172.21.15.28:0/342373415 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740ce890 con 0x7fdd74071a80 2024-03-21T08:41:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.163+0000 7fdd71600640 1 -- 172.21.15.28:0/342373415 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354432239 0 0) 0x7fdd740723a0 con 0x7fdd7415cbb0 2024-03-21T08:41:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd72a00640 1 --2- 172.21.15.28:0/342373415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd74071a80 0x7fdd7406f0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd740af710 tx=0x7fdd6401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ffc16271e6b8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd71600640 1 -- 172.21.15.28:0/342373415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd640ab680 con 0x7fdd74071a80 2024-03-21T08:41:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd71600640 1 -- 172.21.15.28:0/342373415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd640b0070 con 0x7fdd74071a80 2024-03-21T08:41:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd71600640 1 -- 172.21.15.28:0/342373415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd640aba40 con 0x7fdd74071a80 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- 172.21.15.28:0/342373415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd74071a80 msgr2=0x7fdd7406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 --2- 172.21.15.28:0/342373415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd74071a80 0x7fdd7406f0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd740af710 tx=0x7fdd6401c460 comp rx=0 tx=0).stop 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- 172.21.15.28:0/342373415 shutdown_connections 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 --2- 172.21.15.28:0/342373415 >> v2:172.21.15.151:3300/0 conn(0x7fdd740cde40 0x7fdd74071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 --2- 172.21.15.28:0/342373415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd74071a80 0x7fdd7406f0a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- 172.21.15.28:0/342373415 >> 172.21.15.28:0/342373415 conn(0x7fdd740674f0 msgr2=0x7fdd7406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- 172.21.15.28:0/342373415 shutdown_connections 2024-03-21T08:41:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- 172.21.15.28:0/342373415 wait complete. 2024-03-21T08:41:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 Processor -- start 2024-03-21T08:41:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.167+0000 7fdd79800640 1 -- start start 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd74071a80 0x7fdd740af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd7415cbb0 0x7fdd7413a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74168f20 con 0x7fdd740cde40 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd740ce220 con 0x7fdd74071a80 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd741690a0 con 0x7fdd7415cbb0 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35506/0 (socket says 172.21.15.28:35506) 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 -- 172.21.15.28:0/1710435301 learned_addr learned my addr 172.21.15.28:0/1710435301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72a00640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd74071a80 0x7fdd740af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd73400640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd7415cbb0 0x7fdd7413a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 -- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd7415cbb0 msgr2=0x7fdd7413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd7415cbb0 0x7fdd7413a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 -- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd74071a80 msgr2=0x7fdd740af280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd74071a80 0x7fdd740af280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 -- 172.21.15.28:0/1710435301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7415b5a0 con 0x7fdd740cde40 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd72000640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000e930 tx=0x7fdd6000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60017070 con 0x7fdd740cde40 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6000ca00 con 0x7fdd740cde40 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000cdb0 con 0x7fdd740cde40 2024-03-21T08:41:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7413f140 con 0x7fdd740cde40 2024-03-21T08:41:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7413f700 con 0x7fdd740cde40 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.171+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7413fa40 con 0x7fdd740cde40 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.175+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fdd6001a070 con 0x7fdd740cde40 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.175+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 0x7fdd54069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.175+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fdd60085fd0 con 0x7fdd740cde40 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.175+0000 7fdd72a00640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 0x7fdd54069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.175+0000 7fdd72a00640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 0x7fdd54069fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd640af000 tx=0x7fdd640a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.179+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdd7413fa40 con 0x7fdd740cde40 2024-03-21T08:41:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:09.339+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdd74073430 con 0x7fdd740cde40 2024-03-21T08:41:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.011+0000 7fdd6aa00640 1 -- 172.21.15.28:0/1710435301 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7fdd60056310 con 0x7fdd740cde40 2024-03-21T08:41:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-03-21T08:41:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 msgr2=0x7fdd54069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 0x7fdd54069fe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd640af000 tx=0x7fdd640a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 msgr2=0x7fdd7413eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000e930 tx=0x7fdd6000edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 shutdown_connections 2024-03-21T08:41:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd54067b20 0x7fdd54069fe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd7415cbb0 0x7fdd7413a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd740cde40 0x7fdd7413eaf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 --2- 172.21.15.28:0/1710435301 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd74071a80 0x7fdd740af280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.015+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 >> 172.21.15.28:0/1710435301 conn(0x7fdd740674f0 msgr2=0x7fdd74071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.019+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 shutdown_connections 2024-03-21T08:41:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.019+0000 7fdd79800640 1 -- 172.21.15.28:0/1710435301 wait complete. 2024-03-21T08:41:11.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-21T08:41:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.195+0000 7f828fc00640 1 Processor -- start 2024-03-21T08:41:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 -- start start 2024-03-21T08:41:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828809a330 0x7f828806d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82880cfd50 0x7f82880d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880d1600 con 0x7f828815e050 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82880ccab0 con 0x7f828809a330 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82880cc180 con 0x7f82880cfd50 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828809a330 0x7f828806d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f828809a330 0x7f828806d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53082/0 (socket says 172.21.15.28:53082) 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 -- 172.21.15.28:0/2524692409 learned_addr learned my addr 172.21.15.28:0/2524692409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82880cfd50 0x7f82880d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f828d200640 1 --1- 172.21.15.28:0/2524692409 >> v1:172.21.15.151:6789/0 conn(0x7f828815e050 0x7f8288099c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:44712/0 (socket says 172.21.15.28:44712) 2024-03-21T08:41:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 -- 172.21.15.28:0/2524692409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82880cfd50 msgr2=0x7f82880d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 --2- 172.21.15.28:0/2524692409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82880cfd50 0x7f82880d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 -- 172.21.15.28:0/2524692409 >> v1:172.21.15.151:6789/0 conn(0x7f828815e050 legacy=0x7f8288099c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.199+0000 7f827fe00640 1 -- 172.21.15.28:0/2524692409 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880cc460 con 0x7f828809a330 2024-03-21T08:41:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f827fe00640 1 --2- 172.21.15.28:0/2524692409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828809a330 0x7f828806d4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8270009870 tx=0x7f827001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a3e610761ca1bf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f8287e00640 1 -- 172.21.15.28:0/2524692409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82700aa650 con 0x7f828809a330 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f8287e00640 1 -- 172.21.15.28:0/2524692409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82700af070 con 0x7f828809a330 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f8287e00640 1 -- 172.21.15.28:0/2524692409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82700aaa10 con 0x7f828809a330 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- 172.21.15.28:0/2524692409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828809a330 msgr2=0x7f828806d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 --2- 172.21.15.28:0/2524692409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828809a330 0x7f828806d4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8270009870 tx=0x7f827001c4d0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- 172.21.15.28:0/2524692409 shutdown_connections 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 --2- 172.21.15.28:0/2524692409 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82880cfd50 0x7f82880d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 --2- 172.21.15.28:0/2524692409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828809a330 0x7f828806d4f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- 172.21.15.28:0/2524692409 >> 172.21.15.28:0/2524692409 conn(0x7f82880b34f0 msgr2=0x7f82880b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- 172.21.15.28:0/2524692409 shutdown_connections 2024-03-21T08:41:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- 172.21.15.28:0/2524692409 wait complete. 2024-03-21T08:41:11.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 Processor -- start 2024-03-21T08:41:11.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.203+0000 7f828fc00640 1 -- start start 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82880cfd50 0x7f82880b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 0x7f82881519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880d1300 con 0x7f828815e050 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880d1180 con 0x7f82880cfd50 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8288070950 con 0x7f828809a330 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828d200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 0x7f82881519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828d200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54744/0 (socket says 172.21.15.28:54744) 2024-03-21T08:41:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828d200640 1 -- 172.21.15.28:0/1314620070 learned_addr learned my addr 172.21.15.28:0/1314620070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f827fe00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82880cfd50 0x7f82880b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 -- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 msgr2=0x7f8288155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 -- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82880cfd50 msgr2=0x7f82880b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82880cfd50 0x7f82880b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 -- 172.21.15.28:0/1314620070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880700b0 con 0x7f828815e050 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828d200640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828dc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 0x7f82881519d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f828000efc0 tx=0x7f828000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280018020 con 0x7f828815e050 2024-03-21T08:41:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8280014480 con 0x7f828815e050 2024-03-21T08:41:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280014870 con 0x7f828815e050 2024-03-21T08:41:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8288070390 con 0x7f828815e050 2024-03-21T08:41:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828815b2a0 con 0x7f828815e050 2024-03-21T08:41:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.207+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8288068110 con 0x7f828815e050 2024-03-21T08:41:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.211+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f828001b020 con 0x7f828815e050 2024-03-21T08:41:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.211+0000 7f8285600640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 0x7f8264069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.211+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8280086f90 con 0x7f828815e050 2024-03-21T08:41:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.211+0000 7f828d200640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 0x7f8264069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:11.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.211+0000 7f828d200640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 0x7f8264069fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8278007fd0 tx=0x7f8278007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.215+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8280022030 con 0x7f828815e050 2024-03-21T08:41:11.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:11.379+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f828806c350 con 0x7f828815e050 2024-03-21T08:41:11.971 INFO:tasks.ceph.mds.d.smithi151.stderr:2024-03-21T08:41:11.968+0000 7fe4e5c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:41:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.018+0000 7f8285600640 1 -- 172.21.15.28:0/1314620070 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8280010040 con 0x7f828815e050 2024-03-21T08:41:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-03-21T08:41:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 msgr2=0x7f8264069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 0x7f8264069fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8278007fd0 tx=0x7f8278007f20 comp rx=0 tx=0).stop 2024-03-21T08:41:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 msgr2=0x7f82881519d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 0x7f82881519d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f828000efc0 tx=0x7f828000c490 comp rx=0 tx=0).stop 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 shutdown_connections 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f828815e050 0x7f82881519d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82880cfd50 0x7f82880b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8264067b20 0x7f8264069fe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 --2- 172.21.15.28:0/1314620070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f828809a330 0x7f8288155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 >> 172.21.15.28:0/1314620070 conn(0x7f82880b34f0 msgr2=0x7f828806dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 shutdown_connections 2024-03-21T08:41:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.022+0000 7f828fc00640 1 -- 172.21.15.28:0/1314620070 wait complete. 2024-03-21T08:41:12.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-21T08:41:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 Processor -- start 2024-03-21T08:41:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 -- start start 2024-03-21T08:41:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9400b4420 0x7fb940071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb9400b4800 0x7fb940071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94015bf90 con 0x7fb9400ccbf0 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94015b1d0 con 0x7fb9400b4800 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb947c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94015b4b0 con 0x7fb9400b4420 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb945200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9400b4420 0x7fb940071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb9400b4800 0x7fb940071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb9400b4800 0x7fb940071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35784/0 (socket says 172.21.15.28:35784) 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 -- 172.21.15.28:0/2966466 learned_addr learned my addr 172.21.15.28:0/2966466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 -- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9400b4420 msgr2=0x7fb940071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 --2- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9400b4420 0x7fb940071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 -- 172.21.15.28:0/2966466 >> v1:172.21.15.151:6789/0 conn(0x7fb9400ccbf0 legacy=0x7fb9400cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb937e00640 1 -- 172.21.15.28:0/2966466 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94015b790 con 0x7fb9400b4800 2024-03-21T08:41:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.214+0000 7fb93fe00640 1 -- 172.21.15.28:0/2966466 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986977167 0 0) 0x7fb94015bf90 con 0x7fb9400ccbf0 2024-03-21T08:41:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb937e00640 1 --2- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400b4800 0x7fb940071fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb928009870 tx=0x7fb92801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71aba68fb1db50d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb93fe00640 1 -- 172.21.15.28:0/2966466 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280ab430 con 0x7fb9400b4800 2024-03-21T08:41:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb93fe00640 1 -- 172.21.15.28:0/2966466 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9280b0070 con 0x7fb9400b4800 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb93fe00640 1 -- 172.21.15.28:0/2966466 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280ab7f0 con 0x7fb9400b4800 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 -- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400b4800 msgr2=0x7fb940071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 --2- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400b4800 0x7fb940071fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb928009870 tx=0x7fb92801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 -- 172.21.15.28:0/2966466 shutdown_connections 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 --2- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400b4800 0x7fb940071fc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 --2- 172.21.15.28:0/2966466 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9400b4420 0x7fb940071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 -- 172.21.15.28:0/2966466 >> 172.21.15.28:0/2966466 conn(0x7fb9400674f0 msgr2=0x7fb94006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 -- 172.21.15.28:0/2966466 shutdown_connections 2024-03-21T08:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 -- 172.21.15.28:0/2966466 wait complete. 2024-03-21T08:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.218+0000 7fb947c00640 1 Processor -- start 2024-03-21T08:41:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- start start 2024-03-21T08:41:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9400b4420 0x7fb940151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9401486c0 0x7fb940149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400b36b0 con 0x7fb9400b4420 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400b3530 con 0x7fb9400ccbf0 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400b3830 con 0x7fb9401486c0 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb945200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9400b4420 0x7fb940151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35796/0 (socket says 172.21.15.28:35796) 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 -- 172.21.15.28:0/4121528148 learned_addr learned my addr 172.21.15.28:0/4121528148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb945c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9401486c0 0x7fb940149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 -- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9401486c0 msgr2=0x7fb940149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9401486c0 0x7fb940149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 -- 172.21.15.28:0/4121528148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9400b4420 msgr2=0x7fb940151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9400b4420 0x7fb940151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 -- 172.21.15.28:0/4121528148 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94014a0b0 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb937e00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb928009870 tx=0x7fb92801ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928004020 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9280041c0 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928003620 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9401f9fc0 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9401fa580 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.222+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb940073430 con 0x7fb9400ccbf0 2024-03-21T08:41:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.226+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb9280037c0 con 0x7fb9400ccbf0 2024-03-21T08:41:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.226+0000 7fb93d600640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 0x7fb91c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.226+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb928120520 con 0x7fb9400ccbf0 2024-03-21T08:41:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.226+0000 7fb945200640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 0x7fb91c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.226+0000 7fb945200640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 0x7fb91c069fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb930009ac0 tx=0x7fb930009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.230+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb928016150 con 0x7fb9400ccbf0 2024-03-21T08:41:12.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:12.434+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fb94004f4f0 con 0x7fb9400ccbf0 2024-03-21T08:41:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.022+0000 7fb93d600640 1 -- 172.21.15.28:0/4121528148 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb9280f0640 con 0x7fb9400ccbf0 2024-03-21T08:41:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-03-21T08:41:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.022+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 msgr2=0x7fb91c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.022+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 0x7fb91c069fe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb930009ac0 tx=0x7fb930009190 comp rx=0 tx=0).stop 2024-03-21T08:41:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.022+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 msgr2=0x7fb940148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.022+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb928009870 tx=0x7fb92801ace0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 shutdown_connections 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb91c067b20 0x7fb91c069fe0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb9401486c0 0x7fb940149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb9400ccbf0 0x7fb940148180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 --2- 172.21.15.28:0/4121528148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9400b4420 0x7fb940151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 >> 172.21.15.28:0/4121528148 conn(0x7fb9400674f0 msgr2=0x7fb94015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 shutdown_connections 2024-03-21T08:41:13.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.026+0000 7fb947c00640 1 -- 172.21.15.28:0/4121528148 wait complete. 2024-03-21T08:41:13.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.190+0000 7fb2c8600640 1 Processor -- start 2024-03-21T08:41:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.190+0000 7fb2c8600640 1 -- start start 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c015e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2c015ba70 0x7fb2c00ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 0x7fb2c0071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c00cedb0 con 0x7fb2c015ba70 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c015bed0 con 0x7fb2c0071980 2024-03-21T08:41:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c015ebf0 con 0x7fb2c00ce490 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2c015ba70 0x7fb2c00ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2c015ba70 0x7fb2c00ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53104/0 (socket says 172.21.15.28:53104) 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 -- 172.21.15.28:0/4192440621 learned_addr learned my addr 172.21.15.28:0/4192440621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c6600640 1 --2- 172.21.15.28:0/4192440621 >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 0x7fb2c0071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 -- 172.21.15.28:0/4192440621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 msgr2=0x7fb2c015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 -- 172.21.15.28:0/4192440621 >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 msgr2=0x7fb2c0071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 --2- 172.21.15.28:0/4192440621 >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 0x7fb2c0071dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c5200640 1 -- 172.21.15.28:0/4192440621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c015eed0 con 0x7fb2c015ba70 2024-03-21T08:41:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.194+0000 7fb2c6600640 1 --2- 172.21.15.28:0/4192440621 >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 0x7fb2c0071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c5200640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c00ce870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00b5a00 tx=0x7fb2ac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef86f383cba2b7fd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2bfe00640 1 -- 172.21.15.28:0/4192440621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0b0070 con 0x7fb2c015ba70 2024-03-21T08:41:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2bfe00640 1 -- 172.21.15.28:0/4192440621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac0ab600 con 0x7fb2c015ba70 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2bfe00640 1 -- 172.21.15.28:0/4192440621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0ab9d0 con 0x7fb2c015ba70 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- 172.21.15.28:0/4192440621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 msgr2=0x7fb2c00ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c00ce870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00b5a00 tx=0x7fb2ac01c490 comp rx=0 tx=0).stop 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- 172.21.15.28:0/4192440621 shutdown_connections 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 --2- 172.21.15.28:0/4192440621 >> v2:172.21.15.151:3300/0 conn(0x7fb2c0071980 0x7fb2c0071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c00ce870 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 --2- 172.21.15.28:0/4192440621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- 172.21.15.28:0/4192440621 >> 172.21.15.28:0/4192440621 conn(0x7fb2c00cc230 msgr2=0x7fb2c00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- 172.21.15.28:0/4192440621 shutdown_connections 2024-03-21T08:41:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- 172.21.15.28:0/4192440621 wait complete. 2024-03-21T08:41:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 Processor -- start 2024-03-21T08:41:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.198+0000 7fb2c8600640 1 -- start start 2024-03-21T08:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0071980 0x7fb2c0149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c0153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0155390 con 0x7fb2c015ba70 2024-03-21T08:41:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0072720 con 0x7fb2c0071980 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00b4b90 con 0x7fb2c00ce490 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0071980 0x7fb2c0149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53112/0 (socket says 172.21.15.28:53112) 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 -- 172.21.15.28:0/3593213877 learned_addr learned my addr 172.21.15.28:0/3593213877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c5200640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c0153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 -- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 msgr2=0x7fb2c0153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c0153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 -- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0071980 msgr2=0x7fb2c0149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0071980 0x7fb2c0149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 -- 172.21.15.28:0/3593213877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0149ba0 con 0x7fb2c015ba70 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c6600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007e10 tx=0x7fb2b800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b801a070 con 0x7fb2c015ba70 2024-03-21T08:41:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b8003ba0 con 0x7fb2c015ba70 2024-03-21T08:41:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8004030 con 0x7fb2c015ba70 2024-03-21T08:41:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c014af10 con 0x7fb2c015ba70 2024-03-21T08:41:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c014b4d0 con 0x7fb2c015ba70 2024-03-21T08:41:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c004f410 con 0x7fb2c015ba70 2024-03-21T08:41:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.202+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb2b801d070 con 0x7fb2c015ba70 2024-03-21T08:41:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.206+0000 7fb2bd600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 0x7fb2a006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.206+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb2b8086410 con 0x7fb2c015ba70 2024-03-21T08:41:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.206+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 0x7fb2a006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.206+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 0x7fb2a006a010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ae7a0 tx=0x7fb2b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.210+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb2b8010070 con 0x7fb2c015ba70 2024-03-21T08:41:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.402+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2c00b12c0 con 0x7fb2c015ba70 2024-03-21T08:41:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2bd600640 1 -- 172.21.15.28:0/3593213877 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4976 (secure 0 0 0) 0x7fb2b8056480 con 0x7fb2c015ba70 2024-03-21T08:41:13.408 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:13.408 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:12.440098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 msgr2=0x7fb2a006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 0x7fb2a006a010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ae7a0 tx=0x7fb2b4009190 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 msgr2=0x7fb2c0153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8007e10 tx=0x7fb2b800d4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 shutdown_connections 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c015ba70 0x7fb2c0153fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b50 0x7fb2a006a010 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2c00ce490 0x7fb2c0153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 --2- 172.21.15.28:0/3593213877 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0071980 0x7fb2c0149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 >> 172.21.15.28:0/3593213877 conn(0x7fb2c00cc230 msgr2=0x7fb2c014d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 shutdown_connections 2024-03-21T08:41:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.406+0000 7fb2c8600640 1 -- 172.21.15.28:0/3593213877 wait complete. 2024-03-21T08:41:13.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:13.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 Processor -- start 2024-03-21T08:41:13.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 -- start start 2024-03-21T08:41:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f40d40cd380 0x7f40d40b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d415bf00 con 0x7f40d40b5520 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d40cdad0 con 0x7f40d40af060 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.566+0000 7f40da200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d40b3a30 con 0x7f40d40cd380 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40cbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f40d40b5520 0x7f40d415c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50948/0 (socket says 172.21.15.28:50948) 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40cbe00640 1 -- 172.21.15.28:0/75278164 learned_addr learned my addr 172.21.15.28:0/75278164 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d3e00640 1 --2- 172.21.15.28:0/75278164 >> v2:172.21.15.151:3300/0 conn(0x7f40d40cd380 0x7f40d40b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d3e00640 1 -- 172.21.15.28:0/75278164 >> v1:172.21.15.151:6789/0 conn(0x7f40d40af060 legacy=0x7f40d40b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d2a00640 1 -- 172.21.15.28:0/75278164 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243278276 0 0) 0x7f40d415bf00 con 0x7f40d40b5520 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d3e00640 1 -- 172.21.15.28:0/75278164 >> v1:172.21.15.28:6789/0 conn(0x7f40d40b5520 legacy=0x7f40d415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d3e00640 1 -- 172.21.15.28:0/75278164 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d40cddb0 con 0x7f40d40cd380 2024-03-21T08:41:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d3e00640 1 --2- 172.21.15.28:0/75278164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40cd380 0x7f40d40b34f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009d90 tx=0x7f40cc01c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6fa6175045d027 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d2a00640 1 -- 172.21.15.28:0/75278164 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0a66d0 con 0x7f40d40cd380 2024-03-21T08:41:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d2a00640 1 -- 172.21.15.28:0/75278164 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc0a6870 con 0x7f40d40cd380 2024-03-21T08:41:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.570+0000 7f40d2a00640 1 -- 172.21.15.28:0/75278164 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0a6c40 con 0x7f40d40cd380 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- 172.21.15.28:0/75278164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40cd380 msgr2=0x7f40d40b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 --2- 172.21.15.28:0/75278164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40cd380 0x7f40d40b34f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009d90 tx=0x7f40cc01c790 comp rx=0 tx=0).stop 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- 172.21.15.28:0/75278164 shutdown_connections 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 --2- 172.21.15.28:0/75278164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40cd380 0x7f40d40b34f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- 172.21.15.28:0/75278164 >> 172.21.15.28:0/75278164 conn(0x7f40d40cc230 msgr2=0x7f40d40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- 172.21.15.28:0/75278164 shutdown_connections 2024-03-21T08:41:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- 172.21.15.28:0/75278164 wait complete. 2024-03-21T08:41:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 Processor -- start 2024-03-21T08:41:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- start start 2024-03-21T08:41:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40b5520 0x7f40d414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d4151350 0x7f40d4135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d415d2a0 con 0x7f40d40cd380 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d415d120 con 0x7f40d40b5520 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40da200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d415d420 con 0x7f40d4151350 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40d3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40b5520 0x7f40d414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53124/0 (socket says 172.21.15.28:53124) 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 -- 172.21.15.28:0/3929548516 learned_addr learned my addr 172.21.15.28:0/3929548516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40d3e00640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d4151350 0x7f40d4135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 -- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d4151350 msgr2=0x7f40d4135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d4151350 0x7f40d4135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 -- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40b5520 msgr2=0x7f40d414e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40b5520 0x7f40d414e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.574+0000 7f40cbe00640 1 -- 172.21.15.28:0/3929548516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4136a80 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cbe00640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c4d0 tx=0x7f40bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc018070 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc0074e0 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc004a30 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d4136d60 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d414a7b0 con 0x7f40d40cd380 2024-03-21T08:41:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d404f410 con 0x7f40d40cd380 2024-03-21T08:41:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f40bc01b070 con 0x7f40d40cd380 2024-03-21T08:41:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 0x7f40b006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f40bc085550 con 0x7f40d40cd380 2024-03-21T08:41:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.578+0000 7f40d3400640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 0x7f40b006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.582+0000 7f40d3400640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 0x7f40b006a010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f40c4007fd0 tx=0x7f40c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.582+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f40bc055530 con 0x7f40d40cd380 2024-03-21T08:41:13.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.746+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d40723e0 con 0x7f40d40cd380 2024-03-21T08:41:13.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40cb400640 1 -- 172.21.15.28:0/3929548516 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f40bc0595d0 con 0x7f40d40cd380 2024-03-21T08:41:13.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:13.752 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 msgr2=0x7f40b006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 0x7f40b006a010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f40c4007fd0 tx=0x7f40c4009040 comp rx=0 tx=0).stop 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 msgr2=0x7f40d4150e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c4d0 tx=0x7f40bc00c990 comp rx=0 tx=0).stop 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 shutdown_connections 2024-03-21T08:41:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40b0067b50 0x7f40b006a010 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40d4151350 0x7f40d4135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d40cd380 0x7f40d4150e10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 --2- 172.21.15.28:0/3929548516 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40d40b5520 0x7f40d414e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 >> 172.21.15.28:0/3929548516 conn(0x7f40d40cc230 msgr2=0x7f40d40b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 shutdown_connections 2024-03-21T08:41:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.750+0000 7f40da200640 1 -- 172.21.15.28:0/3929548516 wait complete. 2024-03-21T08:41:13.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:13.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.910+0000 7fd289400640 1 Processor -- start 2024-03-21T08:41:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 -- start start 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd2840cdbf0 0x7fd2840cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28406b170 0x7fd284070a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840ce510 con 0x7fd2840cd3a0 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28406b6e0 con 0x7fd28406b170 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd289400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28406b9c0 con 0x7fd2840cdbf0 2024-03-21T08:41:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd27b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28406b170 0x7fd284070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd288a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd2840cd3a0 0x7fd28415c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50964/0 (socket says 172.21.15.28:50964) 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd288a00640 1 -- 172.21.15.28:0/952711612 learned_addr learned my addr 172.21.15.28:0/952711612 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd283e00640 1 --2- 172.21.15.28:0/952711612 >> v2:172.21.15.151:3300/0 conn(0x7fd2840cdbf0 0x7fd2840cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd27b400640 1 -- 172.21.15.28:0/952711612 >> v2:172.21.15.151:3300/0 conn(0x7fd2840cdbf0 msgr2=0x7fd2840cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd27b400640 1 --2- 172.21.15.28:0/952711612 >> v2:172.21.15.151:3300/0 conn(0x7fd2840cdbf0 0x7fd2840cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd27b400640 1 -- 172.21.15.28:0/952711612 >> v1:172.21.15.28:6789/0 conn(0x7fd2840cd3a0 legacy=0x7fd28415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.914+0000 7fd27b400640 1 -- 172.21.15.28:0/952711612 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284070220 con 0x7fd28406b170 2024-03-21T08:41:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd27b400640 1 --2- 172.21.15.28:0/952711612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28406b170 0x7fd284070a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd28406be30 tx=0x7fd26c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca4008bd3b18427 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd283400640 1 -- 172.21.15.28:0/952711612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0b0070 con 0x7fd28406b170 2024-03-21T08:41:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd283400640 1 -- 172.21.15.28:0/952711612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c004230 con 0x7fd28406b170 2024-03-21T08:41:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd283400640 1 -- 172.21.15.28:0/952711612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c003620 con 0x7fd28406b170 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- 172.21.15.28:0/952711612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28406b170 msgr2=0x7fd284070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 --2- 172.21.15.28:0/952711612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28406b170 0x7fd284070a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd28406be30 tx=0x7fd26c01c480 comp rx=0 tx=0).stop 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- 172.21.15.28:0/952711612 shutdown_connections 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 --2- 172.21.15.28:0/952711612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28406b170 0x7fd284070a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 --2- 172.21.15.28:0/952711612 >> v2:172.21.15.151:3300/0 conn(0x7fd2840cdbf0 0x7fd2840cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- 172.21.15.28:0/952711612 >> 172.21.15.28:0/952711612 conn(0x7fd28406e920 msgr2=0x7fd28406ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- 172.21.15.28:0/952711612 shutdown_connections 2024-03-21T08:41:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- 172.21.15.28:0/952711612 wait complete. 2024-03-21T08:41:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 Processor -- start 2024-03-21T08:41:13.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.918+0000 7fd289400640 1 -- start start 2024-03-21T08:41:13.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd28406b170 0x7fd2841faa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2840cd3a0 0x7fd28415ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd284156720 con 0x7fd2840cdbf0 2024-03-21T08:41:13.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841565a0 con 0x7fd2840cd3a0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840cd7b0 con 0x7fd28406b170 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53142/0 (socket says 172.21.15.28:53142) 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 -- 172.21.15.28:0/119260897 learned_addr learned my addr 172.21.15.28:0/119260897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd27b400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2840cd3a0 0x7fd28415ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd283e00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd28406b170 0x7fd2841faa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 -- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd28406b170 msgr2=0x7fd2841faa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd28406b170 0x7fd2841faa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 -- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2840cd3a0 msgr2=0x7fd28415ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2840cd3a0 0x7fd28415ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 -- 172.21.15.28:0/119260897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284140310 con 0x7fd2840cdbf0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd288a00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00cbf0 tx=0x7fd27c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c017070 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c00ace0 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c012600 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd284161000 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd284160a40 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd284075270 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.922+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd27c01a070 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.926+0000 7fd280c00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 0x7fd260069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.926+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd27c085ef0 con 0x7fd2840cdbf0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.926+0000 7fd283e00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 0x7fd260069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.926+0000 7fd283e00640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 0x7fd260069fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd28415f830 tx=0x7fd274009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:13.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:13.930+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd27c055ed0 con 0x7fd2840cdbf0 2024-03-21T08:41:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd284072de0 con 0x7fd2840cdbf0 2024-03-21T08:41:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd280c00640 1 -- 172.21.15.28:0/119260897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7fd27c059f70 con 0x7fd2840cdbf0 2024-03-21T08:41:14.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:14.097 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 msgr2=0x7fd260069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 0x7fd260069fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd28415f830 tx=0x7fd274009190 comp rx=0 tx=0).stop 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 msgr2=0x7fd28413fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00cbf0 tx=0x7fd27c00a430 comp rx=0 tx=0).stop 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 shutdown_connections 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2840cdbf0 0x7fd28413fdd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2840cd3a0 0x7fd28415ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd260067b20 0x7fd260069fe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 --2- 172.21.15.28:0/119260897 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd28406b170 0x7fd2841faa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 >> 172.21.15.28:0/119260897 conn(0x7fd28406e920 msgr2=0x7fd284073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.094+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 shutdown_connections 2024-03-21T08:41:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.098+0000 7fd289400640 1 -- 172.21.15.28:0/119260897 wait complete. 2024-03-21T08:41:14.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 Processor -- start 2024-03-21T08:41:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 -- start start 2024-03-21T08:41:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f83880ce4c0 0x7f83880cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f838806b170 0x7f838806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880ceaa0 con 0x7f838815c680 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838806bb00 con 0x7f838806b170 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f838d400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83880cffe0 con 0x7f83880ce4c0 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f8387200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f838806b170 0x7f838806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f8387200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f838806b170 0x7f838806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53158/0 (socket says 172.21.15.28:53158) 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f837fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f83880ce4c0 0x7f83880cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f8387200640 1 -- 172.21.15.28:0/1378145780 learned_addr learned my addr 172.21.15.28:0/1378145780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f8387200640 1 -- 172.21.15.28:0/1378145780 >> v2:172.21.15.151:3300/0 conn(0x7f83880ce4c0 msgr2=0x7f83880cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.274+0000 7f8387200640 1 --2- 172.21.15.28:0/1378145780 >> v2:172.21.15.151:3300/0 conn(0x7f83880ce4c0 0x7f83880cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8387200640 1 -- 172.21.15.28:0/1378145780 >> v1:172.21.15.151:6789/0 conn(0x7f838815c680 legacy=0x7f838815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8387200640 1 -- 172.21.15.28:0/1378145780 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880ced80 con 0x7f838806b170 2024-03-21T08:41:14.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8387200640 1 --2- 172.21.15.28:0/1378145780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838806b170 0x7f838806b5c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f838806c250 tx=0x7f838001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3fa8cbf3caa05a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8385e00640 1 -- 172.21.15.28:0/1378145780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800a5590 con 0x7f838806b170 2024-03-21T08:41:14.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8385e00640 1 -- 172.21.15.28:0/1378145780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83800a5730 con 0x7f838806b170 2024-03-21T08:41:14.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f8385e00640 1 -- 172.21.15.28:0/1378145780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800a5b00 con 0x7f838806b170 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 -- 172.21.15.28:0/1378145780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838806b170 msgr2=0x7f838806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 --2- 172.21.15.28:0/1378145780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838806b170 0x7f838806b5c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f838806c250 tx=0x7f838001c8d0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 -- 172.21.15.28:0/1378145780 shutdown_connections 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 --2- 172.21.15.28:0/1378145780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838806b170 0x7f838806b5c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 --2- 172.21.15.28:0/1378145780 >> v2:172.21.15.151:3300/0 conn(0x7f83880ce4c0 0x7f83880cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 -- 172.21.15.28:0/1378145780 >> 172.21.15.28:0/1378145780 conn(0x7f8388099a30 msgr2=0x7f8388099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 -- 172.21.15.28:0/1378145780 shutdown_connections 2024-03-21T08:41:14.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.278+0000 7f838d400640 1 -- 172.21.15.28:0/1378145780 wait complete. 2024-03-21T08:41:14.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 Processor -- start 2024-03-21T08:41:14.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- start start 2024-03-21T08:41:14.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f838806b170 0x7f83880c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f838815c680 0x7f8388154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cd500 con 0x7f83880ce4c0 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880cd380 con 0x7f838815c680 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388168aa0 con 0x7f838806b170 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53166/0 (socket says 172.21.15.28:53166) 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 -- 172.21.15.28:0/1277738706 learned_addr learned my addr 172.21.15.28:0/1277738706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f8387200640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f838815c680 0x7f8388154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f8386800640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f838806b170 0x7f83880c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 -- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f838806b170 msgr2=0x7f83880c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f838806b170 0x7f83880c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 -- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f838815c680 msgr2=0x7f8388154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f838815c680 0x7f8388154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 -- 172.21.15.28:0/1277738706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880c3f30 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837fe00640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f837000e870 tx=0x7f837000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370017070 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837000c940 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837000ccf0 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83880b5440 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.282+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838814e940 con 0x7f83880ce4c0 2024-03-21T08:41:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83880b5960 con 0x7f83880ce4c0 2024-03-21T08:41:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f837001a070 con 0x7f83880ce4c0 2024-03-21T08:41:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f837ea00640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 0x7f8364069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8370085ff0 con 0x7f83880ce4c0 2024-03-21T08:41:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f8386800640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 0x7f8364069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.286+0000 7f8386800640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 0x7f8364069fe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8378007fd0 tx=0x7f8378009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.290+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f83880b5960 con 0x7f83880ce4c0 2024-03-21T08:41:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f838804f410 con 0x7f83880ce4c0 2024-03-21T08:41:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f837ea00640 1 -- 172.21.15.28:0/1277738706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f83700560c0 con 0x7f83880ce4c0 2024-03-21T08:41:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:14.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:12.440098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-03-21T08:41:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 msgr2=0x7f8364069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 0x7f8364069fe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8378007fd0 tx=0x7f8378009040 comp rx=0 tx=0).stop 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 msgr2=0x7f8388154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f837000e870 tx=0x7f837000ec70 comp rx=0 tx=0).stop 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 shutdown_connections 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f838815c680 0x7f8388154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83880ce4c0 0x7f8388154710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8364067b20 0x7f8364069fe0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 --2- 172.21.15.28:0/1277738706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f838806b170 0x7f83880c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 >> 172.21.15.28:0/1277738706 conn(0x7f8388099a30 msgr2=0x7f83880b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 shutdown_connections 2024-03-21T08:41:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.490+0000 7f838d400640 1 -- 172.21.15.28:0/1277738706 wait complete. 2024-03-21T08:41:14.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 Processor -- start 2024-03-21T08:41:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 -- start start 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 0x7f0844099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f08440c5f80 0x7f08440c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08440cdb50 con 0x7f084415f260 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084409a520 con 0x7f08440c5f80 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f084ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08440c5530 con 0x7f0844099b80 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f0843e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f084415f260 0x7f08440cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50974/0 (socket says 172.21.15.28:50974) 2024-03-21T08:41:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f0843e00640 1 -- 172.21.15.28:0/3479726866 learned_addr learned my addr 172.21.15.28:0/3479726866 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f083b400640 1 --2- 172.21.15.28:0/3479726866 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 0x7f0844099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f0848e00640 1 --2- 172.21.15.28:0/3479726866 >> v2:172.21.15.151:3300/0 conn(0x7f08440c5f80 0x7f08440c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f0843400640 1 -- 172.21.15.28:0/3479726866 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626599264 0 0) 0x7f08440cdb50 con 0x7f084415f260 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f0843400640 1 -- 172.21.15.28:0/3479726866 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0828003530 con 0x7f084415f260 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f083b400640 1 -- 172.21.15.28:0/3479726866 >> v2:172.21.15.151:3300/0 conn(0x7f08440c5f80 msgr2=0x7f08440c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f083b400640 1 --2- 172.21.15.28:0/3479726866 >> v2:172.21.15.151:3300/0 conn(0x7f08440c5f80 0x7f08440c63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f083b400640 1 -- 172.21.15.28:0/3479726866 >> v1:172.21.15.28:6789/0 conn(0x7f084415f260 legacy=0x7f08440cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.654+0000 7f083b400640 1 -- 172.21.15.28:0/3479726866 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440cde30 con 0x7f0844099b80 2024-03-21T08:41:14.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f083b400640 1 --2- 172.21.15.28:0/3479726866 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 0x7f0844099fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f082c0097c0 tx=0x7f082c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcf1e5c6e685161e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f0843400640 1 -- 172.21.15.28:0/3479726866 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0b1070 con 0x7f0844099b80 2024-03-21T08:41:14.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f0843400640 1 -- 172.21.15.28:0/3479726866 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c0a8050 con 0x7f0844099b80 2024-03-21T08:41:14.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f0843400640 1 -- 172.21.15.28:0/3479726866 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c0ac860 con 0x7f0844099b80 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 -- 172.21.15.28:0/3479726866 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 msgr2=0x7f0844099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 --2- 172.21.15.28:0/3479726866 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 0x7f0844099fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f082c0097c0 tx=0x7f082c01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 -- 172.21.15.28:0/3479726866 shutdown_connections 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 --2- 172.21.15.28:0/3479726866 >> v2:172.21.15.151:3300/0 conn(0x7f08440c5f80 0x7f08440c63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 --2- 172.21.15.28:0/3479726866 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0844099b80 0x7f0844099fe0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 -- 172.21.15.28:0/3479726866 >> 172.21.15.28:0/3479726866 conn(0x7f08440b34d0 msgr2=0x7f08440b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 -- 172.21.15.28:0/3479726866 shutdown_connections 2024-03-21T08:41:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.658+0000 7f084ae00640 1 -- 172.21.15.28:0/3479726866 wait complete. 2024-03-21T08:41:14.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 Processor -- start 2024-03-21T08:41:14.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- start start 2024-03-21T08:41:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08440c5f80 0x7f0844152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f084415f260 0x7f0844156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440cd7a0 con 0x7f0844099b80 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084415f640 con 0x7f084415f260 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0844157530 con 0x7f08440c5f80 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53170/0 (socket says 172.21.15.28:53170) 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 -- 172.21.15.28:0/3055836312 learned_addr learned my addr 172.21.15.28:0/3055836312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:14.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f083b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08440c5f80 0x7f0844152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0848e00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f084415f260 0x7f0844156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 -- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08440c5f80 msgr2=0x7f0844152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08440c5f80 0x7f0844152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 -- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f084415f260 msgr2=0x7f0844156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f084415f260 0x7f0844156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 -- 172.21.15.28:0/3055836312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08441fada0 con 0x7f0844099b80 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0843e00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f083400c480 tx=0x7f083400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834017070 con 0x7f0844099b80 2024-03-21T08:41:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0834005cf0 con 0x7f0844099b80 2024-03-21T08:41:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834007290 con 0x7f0844099b80 2024-03-21T08:41:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08441fa500 con 0x7f0844099b80 2024-03-21T08:41:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08441faac0 con 0x7f0844099b80 2024-03-21T08:41:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.662+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084404f410 con 0x7f0844099b80 2024-03-21T08:41:14.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.666+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f083401a070 con 0x7f0844099b80 2024-03-21T08:41:14.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.666+0000 7f0840c00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 0x7f0820069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:14.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.666+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0834085660 con 0x7f0844099b80 2024-03-21T08:41:14.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.666+0000 7f083b400640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 0x7f0820069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:14.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.666+0000 7f083b400640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 0x7f0820069fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f082c0ae000 tx=0x7f082c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.670+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0834055650 con 0x7f0844099b80 2024-03-21T08:41:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.834+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08440c52c0 con 0x7f0844099b80 2024-03-21T08:41:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.834+0000 7f0840c00640 1 -- 172.21.15.28:0/3055836312 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f08340596f0 con 0x7f0844099b80 2024-03-21T08:41:14.839 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:14.839 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 msgr2=0x7f0820069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 0x7f0820069fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f082c0ae000 tx=0x7f082c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 msgr2=0x7f08441f9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f083400c480 tx=0x7f083400c940 comp rx=0 tx=0).stop 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 shutdown_connections 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f084415f260 0x7f0844156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0820067b20 0x7f0820069fe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08440c5f80 0x7f0844152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 --2- 172.21.15.28:0/3055836312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0844099b80 0x7f08441f9f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 >> 172.21.15.28:0/3055836312 conn(0x7f08440b34d0 msgr2=0x7f0844073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 shutdown_connections 2024-03-21T08:41:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:14.838+0000 7f084ae00640 1 -- 172.21.15.28:0/3055836312 wait complete. 2024-03-21T08:41:15.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-21T08:41:07.397365+0000', 'modified': '2024-03-21T08:41:12.440098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.151:6836/4096075958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6834', 'nonce': 4096075958}, {'type': 'v1', 'addr': '172.21.15.151:6836', 'nonce': 4096075958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-03-21T08:41:15.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-21T08:41:15.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-21T08:41:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.402+0000 7f4f53600640 1 Processor -- start 2024-03-21T08:41:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.402+0000 7f4f53600640 1 -- start start 2024-03-21T08:41:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f53600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4f4c06f0a0 0x7f4f4c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f53600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c06f500 con 0x7f4f4c15cbb0 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f53600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0ce5b0 con 0x7f4f4c071a80 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f53600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4c0ce380 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4f4c06f0a0 0x7f4f4c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f51600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4f4c15cbb0 0x7f4f4c0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50980/0 (socket says 172.21.15.28:50980) 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f51600640 1 -- 172.21.15.28:0/2455678394 learned_addr learned my addr 172.21.15.28:0/2455678394 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f43e00640 1 -- 172.21.15.28:0/2455678394 >> v1:172.21.15.151:6789/0 conn(0x7f4f4c071a80 legacy=0x7f4f4c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f43e00640 1 -- 172.21.15.28:0/2455678394 >> v1:172.21.15.28:6789/0 conn(0x7f4f4c15cbb0 legacy=0x7f4f4c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f43e00640 1 -- 172.21.15.28:0/2455678394 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0ce890 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2455678394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c0cde40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f3401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912408cae4eccc1a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f4be00640 1 -- 172.21.15.28:0/2455678394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340b0070 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f4be00640 1 -- 172.21.15.28:0/2455678394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f340ab600 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.406+0000 7f4f4be00640 1 -- 172.21.15.28:0/2455678394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340ab9d0 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- 172.21.15.28:0/2455678394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c06f0a0 msgr2=0x7f4f4c0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 --2- 172.21.15.28:0/2455678394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c0cde40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f3401c410 comp rx=0 tx=0).stop 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- 172.21.15.28:0/2455678394 shutdown_connections 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 --2- 172.21.15.28:0/2455678394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c0cde40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- 172.21.15.28:0/2455678394 >> 172.21.15.28:0/2455678394 conn(0x7f4f4c0674f0 msgr2=0x7f4f4c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- 172.21.15.28:0/2455678394 shutdown_connections 2024-03-21T08:41:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- 172.21.15.28:0/2455678394 wait complete. 2024-03-21T08:41:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 Processor -- start 2024-03-21T08:41:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- start start 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c13b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f4c15cbb0 0x7f4f4c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c16a0b0 con 0x7f4f4c071a80 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c169f30 con 0x7f4f4c06f0a0 2024-03-21T08:41:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f53600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c16a230 con 0x7f4f4c15cbb0 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53198/0 (socket says 172.21.15.28:53198) 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 -- 172.21.15.28:0/3376101614 learned_addr learned my addr 172.21.15.28:0/3376101614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f50c00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c13b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f51600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f4c15cbb0 0x7f4f4c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 -- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f4c15cbb0 msgr2=0x7f4f4c158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f4c15cbb0 0x7f4f4c158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.410+0000 7f4f43e00640 1 -- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f4c06f0a0 msgr2=0x7f4f4c13b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f43e00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c13b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f43e00640 1 -- 172.21.15.28:0/3376101614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0afd30 con 0x7f4f4c071a80 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f43e00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f340191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 -- 172.21.15.28:0/3376101614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340b8020 con 0x7f4f4c071a80 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 -- 172.21.15.28:0/3376101614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f340b4630 con 0x7f4f4c071a80 2024-03-21T08:41:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c0affd0 con 0x7f4f4c071a80 2024-03-21T08:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 -- 172.21.15.28:0/3376101614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340b0040 con 0x7f4f4c071a80 2024-03-21T08:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c0b0590 con 0x7f4f4c071a80 2024-03-21T08:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 -- 172.21.15.28:0/3376101614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4f340bb020 con 0x7f4f4c071a80 2024-03-21T08:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 0x7f4f28069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f49600640 1 -- 172.21.15.28:0/3376101614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4f3411fe40 con 0x7f4f4c071a80 2024-03-21T08:41:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.414+0000 7f4f50c00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 0x7f4f28069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 Processor -- start 2024-03-21T08:41:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- start start 2024-03-21T08:41:15.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f50c00640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 0x7f4f28069fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c007fd0 tx=0x7f4f3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 0x560da386bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da386c240 0x560da3821ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da366f680 con 0x560da3821ff0 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da35f6b00 con 0x560da386b920 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da366b720 con 0x560da386c240 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 0x560da386bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53208/0 (socket says 172.21.15.28:53208) 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da386c240 0x560da3821ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 -- 172.21.15.28:0/3348439802 learned_addr learned my addr 172.21.15.28:0/3348439802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 -- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da386c240 msgr2=0x560da3821ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da386c240 0x560da3821ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 -- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 msgr2=0x560da386bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 0x560da386bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 -- 172.21.15.28:0/3348439802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560da3647940 con 0x560da3821ff0 2024-03-21T08:41:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f50c00640 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 0x560da386bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f51600640 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f44002020 tx=0x7f4f4400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb208c245f5b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f41600640 1 -- 172.21.15.28:0/3348439802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44019070 con 0x560da3821ff0 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f41600640 1 -- 172.21.15.28:0/3348439802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400ee50 con 0x560da3821ff0 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f41600640 1 -- 172.21.15.28:0/3348439802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44014680 con 0x560da3821ff0 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- 172.21.15.28:0/3348439802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 msgr2=0x560da38223d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f44002020 tx=0x7f4f4400e530 comp rx=0 tx=0).stop 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- 172.21.15.28:0/3348439802 shutdown_connections 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da38223d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da386c240 0x560da3821ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/3348439802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386b920 0x560da386bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- 172.21.15.28:0/3348439802 >> 172.21.15.28:0/3348439802 conn(0x560da381dfa0 msgr2=0x560da3820390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- 172.21.15.28:0/3348439802 shutdown_connections 2024-03-21T08:41:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- 172.21.15.28:0/3348439802 wait complete. 2024-03-21T08:41:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 -- start start 2024-03-21T08:41:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.418+0000 7f4f56b2c000 1 Processor -- start 2024-03-21T08:41:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da3850d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386c240 0x560da38517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da366e990 con 0x560da386b920 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da3629ae0 con 0x560da386c240 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da3607750 con 0x560da3821ff0 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 10 client.0 _ll_register_callbacks cb 0x560da380cff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da3850d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53218/0 (socket says 172.21.15.28:53218) 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 -- 172.21.15.28:0/2908099146 learned_addr learned my addr 172.21.15.28:0/2908099146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f51600640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386c240 0x560da38517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 -- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 msgr2=0x560da3850d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da3850d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 -- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386c240 msgr2=0x560da38517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386c240 0x560da38517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 -- 172.21.15.28:0/2908099146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560da36b4d80 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f3401c980 tx=0x7f4f340f05c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f70c669ec235332 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340f8780 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4f340ab4c0 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f4400ee50 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f340f0060 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 10 client.4434 fetch_fsmap learned FSMap version 7 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x560da35f6d10 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400ee50 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f4f340b3050 con 0x560da386b920 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 10 client.4434 fetch_fsmap finished waiting for FSMap version 7 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 10 client.4434 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-21T08:41:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 20 client.4434 populate_metadata read hostname 'smithi028' 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da3852090 0x560da387dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 4 client.4434 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 --> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560da360ab60 con 0x560da3852090 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f50c00640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da3852090 0x560da387dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f3a000640 10 client.4434 ms_handle_connect on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.422+0000 7f4f50c00640 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da3852090 0x560da387dea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2156d04c0fafbf77 server_cookie=fb35cf77a6bc3481 in_seq=0 out_seq=0 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f3a000640 1 -- 172.21.15.28:0/2908099146 <== mds.0 v2:172.21.15.151:6834/4096075958 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560da360ab60 con 0x560da3852090 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f3a000640 10 client.4434 handle_command_reply: tid=0 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 client.4434 shutdown 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 msgr2=0x560da3851280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f3401c980 tx=0x7f4f340f05c0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 shutdown_connections 2024-03-21T08:41:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da3852090 0x560da387dea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da386c240 0x560da38517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da386b920 0x560da3851280 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f3a000640 0 client.4434 ms_handle_reset on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2908099146 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da3850d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 >> 172.21.15.28:0/2908099146 conn(0x560da381dfa0 msgr2=0x560da3820c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 shutdown_connections 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2908099146 wait complete. 2024-03-21T08:41:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.426+0000 7f4f56b2c000 20 client.4434 trim_cache size 0 max 16384 2024-03-21T08:41:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 Processor -- start 2024-03-21T08:41:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 -- start start 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 0x560da38217c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da386aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c204d00 con 0x560da3850ed0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c203e50 con 0x560da381dfa0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c141df0 con 0x560da3821ff0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 0x560da38217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f51600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da386aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 0x560da38217c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53230/0 (socket says 172.21.15.28:53230) 2024-03-21T08:41:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f51600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da386aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54842/0 (socket says 172.21.15.28:54842) 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f50c00640 1 -- 172.21.15.28:0/2359538680 learned_addr learned my addr 172.21.15.28:0/2359538680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35934/0 (socket says 172.21.15.28:35934) 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 -- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 msgr2=0x560da386aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da386aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 -- 172.21.15.28:0/2359538680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 msgr2=0x560da38217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 0x560da38217c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.442+0000 7f4f43e00640 1 -- 172.21.15.28:0/2359538680 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560da38261d0 con 0x560da381dfa0 2024-03-21T08:41:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f34097df0 tx=0x7f4f340f2000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6945dee52f43932 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f40c00640 1 -- 172.21.15.28:0/2359538680 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340f8780 con 0x560da381dfa0 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f40c00640 1 -- 172.21.15.28:0/2359538680 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f340ab4c0 con 0x560da381dfa0 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f40c00640 1 -- 172.21.15.28:0/2359538680 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f340ec0a0 con 0x560da381dfa0 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 msgr2=0x560da386a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f34097df0 tx=0x7f4f340f2000 comp rx=0 tx=0).stop 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2359538680 shutdown_connections 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3821ff0 0x560da386aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da386a530 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2359538680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3850ed0 0x560da38217c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2359538680 >> 172.21.15.28:0/2359538680 conn(0x560da3824750 msgr2=0x560da384e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2359538680 shutdown_connections 2024-03-21T08:41:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2359538680 wait complete. 2024-03-21T08:41:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- start start 2024-03-21T08:41:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 Processor -- start 2024-03-21T08:41:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da387f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3850ed0 0x560da387fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da3617a80 con 0x560da3821ff0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c203850 con 0x560da381dfa0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da372c700 con 0x560da3850ed0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f56b2c000 10 client.0 _ll_register_callbacks cb 0x7f4f340efbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da387f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35942/0 (socket says 172.21.15.28:35942) 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 -- 172.21.15.28:0/2719202720 learned_addr learned my addr 172.21.15.28:0/2719202720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da387f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53246/0 (socket says 172.21.15.28:53246) 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f51600640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3850ed0 0x560da387fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 -- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3850ed0 msgr2=0x560da387fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3850ed0 0x560da387fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 -- 172.21.15.28:0/2719202720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 msgr2=0x560da387f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da387f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 -- 172.21.15.28:0/2719202720 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560da363c540 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f50c00640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c02a750 tx=0x7f4f3c001090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b352331ce9497c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.446+0000 7f4f3a000640 10 client.0 ms_handle_connect on v2:172.21.15.151:3300/0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c02a310 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mon.1 v2:172.21.15.151:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4f3c014660 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f340ab4c0 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f3c038050 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 10 client.4408 fetch_fsmap learned FSMap version 7 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x560da36034b0 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mon.1 v2:172.21.15.151:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f340ab4c0 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mon.1 v2:172.21.15.151:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f4f3c03b020 con 0x560da381dfa0 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 10 client.4408 fetch_fsmap finished waiting for FSMap version 7 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 10 client.4408 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-21T08:41:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 20 client.4408 populate_metadata read hostname 'smithi028' 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da386b920 0x560da38d91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 4 client.4408 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 --> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560da367cbc0 con 0x560da386b920 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da386b920 0x560da38d91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f43e00640 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da386b920 0x560da38d91f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c88586084a34c50 server_cookie=b88c892442d01c8b in_seq=0 out_seq=0 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 10 client.4408 ms_handle_connect on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 1 -- 172.21.15.28:0/2719202720 <== mds.0 v2:172.21.15.151:6834/4096075958 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x560da367cbc0 con 0x560da386b920 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 10 client.4408 handle_command_reply: tid=0 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 client.4408 shutdown 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 msgr2=0x560da387f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c02a750 tx=0x7f4f3c001090 comp rx=0 tx=0).stop 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 shutdown_connections 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560da386b920 0x560da38d91f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560da3850ed0 0x560da387fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560da3821ff0 0x560da387f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 --2- 172.21.15.28:0/2719202720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560da381dfa0 0x560da387f2e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f3a000640 0 client.4408 ms_handle_reset on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 >> 172.21.15.28:0/2719202720 conn(0x560da3824750 msgr2=0x560da384ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 shutdown_connections 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 1 -- 172.21.15.28:0/2719202720 wait complete. 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f56b2c000 20 client.4408 trim_cache size 0 max 16384 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "51dba44a-cba1-4b57-a836-26a407968834", 2024-03-21T08:41:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4293, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 7.455573362, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 7, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 25, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 24, 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.486318698 2024-03-21T08:41:15.459 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 msgr2=0x7f4f28069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 0x7f4f28069fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c007fd0 tx=0x7f4f3c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 msgr2=0x7f4f4c0b16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f340191b0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 shutdown_connections 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f28067b20 0x7f4f28069fe0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f4c15cbb0 0x7f4f4c158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c071a80 0x7f4f4c0b16c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 --2- 172.21.15.28:0/3376101614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f4c06f0a0 0x7f4f4c13b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 >> 172.21.15.28:0/3376101614 conn(0x7f4f4c0674f0 msgr2=0x7f4f4c0ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 shutdown_connections 2024-03-21T08:41:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.450+0000 7f4f53600640 1 -- 172.21.15.28:0/3376101614 wait complete. 2024-03-21T08:41:15.473 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T08:41:15.473 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:15.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:41:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.610+0000 7fd1ac600640 1 Processor -- start 2024-03-21T08:41:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.610+0000 7fd1ac600640 1 -- start start 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1ac600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1a4099b80 0x7fd1a409a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1ac600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd1a40cfcb0 0x7fd1a40d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1ac600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40cc1d0 con 0x7fd1a415e0c0 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1ac600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a40d1730 con 0x7fd1a4099b80 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1ac600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a40d1450 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1a4099b80 0x7fd1a409a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1a4099b80 0x7fd1a409a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53262/0 (socket says 172.21.15.28:53262) 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1aa600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd1a40cfcb0 0x7fd1a40d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 -- 172.21.15.28:0/1264337729 learned_addr learned my addr 172.21.15.28:0/1264337729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 -- 172.21.15.28:0/1264337729 >> v2:172.21.15.151:3300/0 conn(0x7fd1a40cfcb0 msgr2=0x7fd1a40d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 --2- 172.21.15.28:0/1264337729 >> v2:172.21.15.151:3300/0 conn(0x7fd1a40cfcb0 0x7fd1a40d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 -- 172.21.15.28:0/1264337729 >> v1:172.21.15.151:6789/0 conn(0x7fd1a415e0c0 legacy=0x7fd1a406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.614+0000 7fd1a9200640 1 -- 172.21.15.28:0/1264337729 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40cc470 con 0x7fd1a4099b80 2024-03-21T08:41:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1a9200640 1 --2- 172.21.15.28:0/1264337729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a4099b80 0x7fd1a409a010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40734d0 tx=0x7fd19001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a9f0796530751a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1a3e00640 1 -- 172.21.15.28:0/1264337729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900b1070 con 0x7fd1a4099b80 2024-03-21T08:41:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1a3e00640 1 -- 172.21.15.28:0/1264337729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1900ac600 con 0x7fd1a4099b80 2024-03-21T08:41:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1a3e00640 1 -- 172.21.15.28:0/1264337729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900ac9d0 con 0x7fd1a4099b80 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- 172.21.15.28:0/1264337729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a4099b80 msgr2=0x7fd1a409a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 --2- 172.21.15.28:0/1264337729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a4099b80 0x7fd1a409a010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40734d0 tx=0x7fd19001c520 comp rx=0 tx=0).stop 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- 172.21.15.28:0/1264337729 shutdown_connections 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 --2- 172.21.15.28:0/1264337729 >> v2:172.21.15.151:3300/0 conn(0x7fd1a40cfcb0 0x7fd1a40d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 --2- 172.21.15.28:0/1264337729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a4099b80 0x7fd1a409a010 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- 172.21.15.28:0/1264337729 >> 172.21.15.28:0/1264337729 conn(0x7fd1a40b34d0 msgr2=0x7fd1a40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- 172.21.15.28:0/1264337729 shutdown_connections 2024-03-21T08:41:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- 172.21.15.28:0/1264337729 wait complete. 2024-03-21T08:41:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 Processor -- start 2024-03-21T08:41:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 -- start start 2024-03-21T08:41:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1a4099b80 0x7fd1a40c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.618+0000 7fd1ac600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 0x7fd1a40c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40d0e70 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40d0ff0 con 0x7fd1a415e0c0 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40d1170 con 0x7fd1a4099b80 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53268/0 (socket says 172.21.15.28:53268) 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 -- 172.21.15.28:0/2241537653 learned_addr learned my addr 172.21.15.28:0/2241537653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1aa600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 0x7fd1a40c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9c00640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1a4099b80 0x7fd1a40c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 -- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1a4099b80 msgr2=0x7fd1a40c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1a4099b80 0x7fd1a40c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 -- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 msgr2=0x7fd1a40c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 0x7fd1a40c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 -- 172.21.15.28:0/2241537653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40c7db0 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1aa600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 0x7fd1a40c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a9200640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd190019130 tx=0x7fd19001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900bc070 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1900b7430 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900b1040 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a40c8090 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a4148e20 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a404f410 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd1900bf070 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.622+0000 7fd1a1600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 0x7fd184069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.626+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd19011f880 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.626+0000 7fd1a9c00640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 0x7fd184069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.626+0000 7fd1a9c00640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 0x7fd184069fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd198001650 tx=0x7fd198009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.626+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd190016110 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.826+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd1a406c3b0 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.826+0000 7fd1a1600640 1 -- 172.21.15.28:0/2241537653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd1900eff40 con 0x7fd1a40cfcb0 2024-03-21T08:41:15.831 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 msgr2=0x7fd184069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 0x7fd184069fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd198001650 tx=0x7fd198009040 comp rx=0 tx=0).stop 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 msgr2=0x7fd1a40c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd190019130 tx=0x7fd19001ad60 comp rx=0 tx=0).stop 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 shutdown_connections 2024-03-21T08:41:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1a415e0c0 0x7fd1a40c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1a40cfcb0 0x7fd1a40c5750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd184067b20 0x7fd184069fe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 --2- 172.21.15.28:0/2241537653 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1a4099b80 0x7fd1a40c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 >> 172.21.15.28:0/2241537653 conn(0x7fd1a40b34d0 msgr2=0x7fd1a40701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 shutdown_connections 2024-03-21T08:41:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:15.830+0000 7fd1ac600640 1 -- 172.21.15.28:0/2241537653 wait complete. 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:41:15.852 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-21T08:41:15.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:15.853 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:15.863 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42394sec preferred_lft 42394sec 2024-03-21T08:41:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:41:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:15.866 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-21T08:41:15.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:15.866 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-21T08:41:15.867 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:15.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:15.979 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:41:15.998 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:41:16.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.001 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:41:16.063 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:41:16.063 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:41:16.063 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:41:16.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:41:16.064 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:41:16.064 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:41:16.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:41:16.065 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:16.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.161 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:41:16.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.171 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-03-21T08:41:16.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.220 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:16.220 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:16.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:16.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-21T08:41:16.220 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:16.301 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-21T08:41:16.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.301 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:16.301 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:16.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-21T08:41:16.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-21T08:41:16.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-21T08:41:16.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-21T08:41:16.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-21T08:41:16.302 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:16.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.456 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:16.456 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:16.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:41:16.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-03-21T08:41:16.456 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:16.495 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-21T08:41:16.495 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:16.496 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-21T08:41:16.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:16.551 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:16.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:16.663 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:16.716 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:16.716 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:16.716 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:16.716 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:16.717 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:16.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.719 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:16.770 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:41:16.771 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-21T08:41:16.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:16.838 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:16.892 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:16.893 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:16.894 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:16.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:16.896 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:16.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-03-21T08:41:16.926+0000 7fb647c33580 -1 init, newargv = 0x5634d0db8910 newargc=15 2024-03-21T08:41:16.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[32102]: starting ceph client 2024-03-21T08:41:16.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[32102]: starting fuse 2024-03-21T08:41:18.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:18.063 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:18.117 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:18.117 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:18.117 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:18.117 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:18.117 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:18.118 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:18.119 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:18.120 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:18.120 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:18.120 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:18.120 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:18.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:18.121 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:18.174 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:41:18.175 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:41:18.175 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:18.176 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-03-21T08:41:18.177 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:41:18.307 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.32102.asok 2024-03-21T08:41:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:41:18.319 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.32102.asok 2024-03-21T08:41:18.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:18.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.32102.asok status 2024-03-21T08:41:18.497 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:41:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:41:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "32102", 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:41:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4337, 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4337 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3374785423 2024-03-21T08:41:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3374785423 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4337 172.21.15.28:0/3374785423", 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "172.21.15.28:0/3374785423", 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:41:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 7, 2024-03-21T08:41:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 25, 2024-03-21T08:41:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:41:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:41:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:41:18.501 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:41:18.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:18.513 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:41:18.513 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:18.524 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:41:18.524 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:18.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:18.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:18.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:18.588 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-21T08:41:18.669 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-03-21T08:41:18.670 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-21T08:41:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:18.671 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:41:18.671 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:41:18.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.902+0000 7efe72a00640 1 Processor -- start 2024-03-21T08:41:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.902+0000 7efe72a00640 1 -- start start 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe72a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7efe6c15e610 0x7efe6c15ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe72a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe6c0b4380 0x7efe6c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe72a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c15ebf0 con 0x7efe6c0ce490 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe72a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c15eed0 con 0x7efe6c0b4380 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe72a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c15f1b0 con 0x7efe6c15e610 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe6c0b4380 0x7efe6c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe6be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efe6c0ce490 0x7efe6c0ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50986/0 (socket says 172.21.15.28:50986) 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe6c0b4380 0x7efe6c0b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53278/0 (socket says 172.21.15.28:53278) 2024-03-21T08:41:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe6be00640 1 -- 172.21.15.28:0/881086888 learned_addr learned my addr 172.21.15.28:0/881086888 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:18.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe63e00640 1 --2- 172.21.15.28:0/881086888 >> v2:172.21.15.151:3300/0 conn(0x7efe6c15e610 0x7efe6c15ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 -- 172.21.15.28:0/881086888 >> v2:172.21.15.151:3300/0 conn(0x7efe6c15e610 msgr2=0x7efe6c15ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:18.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 --2- 172.21.15.28:0/881086888 >> v2:172.21.15.151:3300/0 conn(0x7efe6c15e610 0x7efe6c15ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 -- 172.21.15.28:0/881086888 >> v1:172.21.15.28:6789/0 conn(0x7efe6c0ce490 legacy=0x7efe6c0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:18.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 -- 172.21.15.28:0/881086888 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c15f490 con 0x7efe6c0b4380 2024-03-21T08:41:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.906+0000 7efe70a00640 1 --2- 172.21.15.28:0/881086888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c0b4380 0x7efe6c0b47b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0b5a00 tx=0x7efe6401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b031c6b3e7144df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe6b400640 1 -- 172.21.15.28:0/881086888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640b2070 con 0x7efe6c0b4380 2024-03-21T08:41:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe6b400640 1 -- 172.21.15.28:0/881086888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe640a5760 con 0x7efe6c0b4380 2024-03-21T08:41:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe6b400640 1 -- 172.21.15.28:0/881086888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640a5b30 con 0x7efe6c0b4380 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- 172.21.15.28:0/881086888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c0b4380 msgr2=0x7efe6c0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- 172.21.15.28:0/881086888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c0b4380 0x7efe6c0b47b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0b5a00 tx=0x7efe6401c8a0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- 172.21.15.28:0/881086888 shutdown_connections 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- 172.21.15.28:0/881086888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c0b4380 0x7efe6c0b47b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- 172.21.15.28:0/881086888 >> v2:172.21.15.151:3300/0 conn(0x7efe6c15e610 0x7efe6c15ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- 172.21.15.28:0/881086888 >> 172.21.15.28:0/881086888 conn(0x7efe6c0cc230 msgr2=0x7efe6c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- 172.21.15.28:0/881086888 shutdown_connections 2024-03-21T08:41:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- 172.21.15.28:0/881086888 wait complete. 2024-03-21T08:41:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 Processor -- start 2024-03-21T08:41:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- start start 2024-03-21T08:41:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe6c0b4380 0x7efe6c1420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe6c0ce490 0x7efe6c06f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c167d40 con 0x7efe6c15e610 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c167bc0 con 0x7efe6c0b4380 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe72a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c167ec0 con 0x7efe6c0ce490 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53292/0 (socket says 172.21.15.28:53292) 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe70a00640 1 -- 172.21.15.28:0/814767638 learned_addr learned my addr 172.21.15.28:0/814767638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe6be00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe6c0b4380 0x7efe6c1420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.910+0000 7efe63e00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe6c0ce490 0x7efe6c06f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 -- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe6c0ce490 msgr2=0x7efe6c06f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe6c0ce490 0x7efe6c06f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 -- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe6c0b4380 msgr2=0x7efe6c1420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe6c0b4380 0x7efe6c1420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 -- 172.21.15.28:0/814767638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0ae6e0 con 0x7efe6c15e610 2024-03-21T08:41:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe70a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efe640194b0 tx=0x7efe64019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640b2050 con 0x7efe6c15e610 2024-03-21T08:41:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe640be070 con 0x7efe6c15e610 2024-03-21T08:41:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c06ee00 con 0x7efe6c15e610 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640ad890 con 0x7efe6c15e610 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c06e040 con 0x7efe6c15e610 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c04f410 con 0x7efe6c15e610 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7efe640ada30 con 0x7efe6c15e610 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 0x7efe4806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efe641213c0 con 0x7efe6c15e610 2024-03-21T08:41:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.914+0000 7efe6be00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 0x7efe4806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.918+0000 7efe6be00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 0x7efe4806a010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efe5c000f70 tx=0x7efe5c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:18.922+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe64016520 con 0x7efe6c15e610 2024-03-21T08:41:19.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.082+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe6c072c10 con 0x7efe6c15e610 2024-03-21T08:41:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.082+0000 7efe68c00640 1 -- 172.21.15.28:0/814767638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7efe640f14c0 con 0x7efe6c15e610 2024-03-21T08:41:19.087 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:19.087 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 msgr2=0x7efe4806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 0x7efe4806a010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efe5c000f70 tx=0x7efe5c007e90 comp rx=0 tx=0).stop 2024-03-21T08:41:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 msgr2=0x7efe6c14ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efe640194b0 tx=0x7efe64019640 comp rx=0 tx=0).stop 2024-03-21T08:41:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 shutdown_connections 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe6c15e610 0x7efe6c14ff40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efe48067b50 0x7efe4806a010 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efe6c0ce490 0x7efe6c06f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 --2- 172.21.15.28:0/814767638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efe6c0b4380 0x7efe6c1420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 >> 172.21.15.28:0/814767638 conn(0x7efe6c0cc230 msgr2=0x7efe6c151210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 shutdown_connections 2024-03-21T08:41:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.086+0000 7efe72a00640 1 -- 172.21.15.28:0/814767638 wait complete. 2024-03-21T08:41:19.109 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:19.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 Processor -- start 2024-03-21T08:41:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 -- start start 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff6d0071a80 0x7ff6d006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 0x7ff6d00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00ce5b0 con 0x7ff6d00cde40 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d015c720 con 0x7ff6d0071a80 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6d4c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00ce890 con 0x7ff6d015cbb0 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff6d00cde40 0x7ff6d0071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50994/0 (socket says 172.21.15.28:50994) 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6c7e00640 1 -- 172.21.15.28:0/940211046 learned_addr learned my addr 172.21.15.28:0/940211046 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6cde00640 1 --2- 172.21.15.28:0/940211046 >> v2:172.21.15.151:3300/0 conn(0x7ff6d0071a80 0x7ff6d006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6ce800640 1 --2- 172.21.15.28:0/940211046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 0x7ff6d00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6cd400640 1 -- 172.21.15.28:0/940211046 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331247042 0 0) 0x7ff6d00ce5b0 con 0x7ff6d00cde40 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6cd400640 1 -- 172.21.15.28:0/940211046 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b4003530 con 0x7ff6d00cde40 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6ce800640 1 -- 172.21.15.28:0/940211046 >> v2:172.21.15.151:3300/0 conn(0x7ff6d0071a80 msgr2=0x7ff6d006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6ce800640 1 --2- 172.21.15.28:0/940211046 >> v2:172.21.15.151:3300/0 conn(0x7ff6d0071a80 0x7ff6d006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6ce800640 1 -- 172.21.15.28:0/940211046 >> v1:172.21.15.28:6789/0 conn(0x7ff6d00cde40 legacy=0x7ff6d0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.250+0000 7ff6ce800640 1 -- 172.21.15.28:0/940211046 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d015c3b0 con 0x7ff6d015cbb0 2024-03-21T08:41:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6ce800640 1 --2- 172.21.15.28:0/940211046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 0x7ff6d00cc4b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009d90 tx=0x7ff6c801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=970545925a227a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6cd400640 1 -- 172.21.15.28:0/940211046 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80b2070 con 0x7ff6d015cbb0 2024-03-21T08:41:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6cd400640 1 -- 172.21.15.28:0/940211046 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c80a5980 con 0x7ff6d015cbb0 2024-03-21T08:41:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6cd400640 1 -- 172.21.15.28:0/940211046 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80a5d50 con 0x7ff6d015cbb0 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 -- 172.21.15.28:0/940211046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 msgr2=0x7ff6d00cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/940211046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 0x7ff6d00cc4b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009d90 tx=0x7ff6c801c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 -- 172.21.15.28:0/940211046 shutdown_connections 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/940211046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d015cbb0 0x7ff6d00cc4b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/940211046 >> v2:172.21.15.151:3300/0 conn(0x7ff6d0071a80 0x7ff6d006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 -- 172.21.15.28:0/940211046 >> 172.21.15.28:0/940211046 conn(0x7ff6d00674f0 msgr2=0x7ff6d006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 -- 172.21.15.28:0/940211046 shutdown_connections 2024-03-21T08:41:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.254+0000 7ff6d4c00640 1 -- 172.21.15.28:0/940211046 wait complete. 2024-03-21T08:41:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 Processor -- start 2024-03-21T08:41:19.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- start start 2024-03-21T08:41:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 0x7ff6d0141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 0x7ff6d015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0168f20 con 0x7ff6d015cbb0 2024-03-21T08:41:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00ce220 con 0x7ff6d0071a80 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d01690a0 con 0x7ff6d00cde40 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6cde00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 0x7ff6d0141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 0x7ff6d015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53304/0 (socket says 172.21.15.28:53304) 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 -- 172.21.15.28:0/648969101 learned_addr learned my addr 172.21.15.28:0/648969101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 -- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 msgr2=0x7ff6d015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 0x7ff6d015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 -- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 msgr2=0x7ff6d0141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 0x7ff6d0141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 -- 172.21.15.28:0/648969101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d00ac470 con 0x7ff6d015cbb0 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6cde00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 0x7ff6d0141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6c7e00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 0x7ff6d015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6ce800640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80b0040 tx=0x7ff6c8019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80b2050 con 0x7ff6d015cbb0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c80be070 con 0x7ff6d015cbb0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80ada70 con 0x7ff6d015cbb0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d00ac750 con 0x7ff6d015cbb0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d00a8c60 con 0x7ff6d015cbb0 2024-03-21T08:41:19.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.258+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d0073430 con 0x7ff6d015cbb0 2024-03-21T08:41:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.262+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff6c80a5800 con 0x7ff6d015cbb0 2024-03-21T08:41:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.262+0000 7ff6c6000640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 0x7ff6ac069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.262+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff6c8121390 con 0x7ff6d015cbb0 2024-03-21T08:41:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.262+0000 7ff6cde00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 0x7ff6ac069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.262+0000 7ff6cde00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 0x7ff6ac069fe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009f50 tx=0x7ff6c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.266+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff6c8016560 con 0x7ff6d015cbb0 2024-03-21T08:41:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6d004f4f0 con 0x7ff6d015cbb0 2024-03-21T08:41:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6c6000640 1 -- 172.21.15.28:0/648969101 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6c80f12e0 con 0x7ff6d015cbb0 2024-03-21T08:41:19.432 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 msgr2=0x7ff6ac069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 0x7ff6ac069fe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009f50 tx=0x7ff6c0009190 comp rx=0 tx=0).stop 2024-03-21T08:41:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 msgr2=0x7ff6d015a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80b0040 tx=0x7ff6c8019680 comp rx=0 tx=0).stop 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 shutdown_connections 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d015cbb0 0x7ff6d015a6c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff6ac067b20 0x7ff6ac069fe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6d00cde40 0x7ff6d015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 --2- 172.21.15.28:0/648969101 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6d0071a80 0x7ff6d0141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 >> 172.21.15.28:0/648969101 conn(0x7ff6d00674f0 msgr2=0x7ff6d00b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 shutdown_connections 2024-03-21T08:41:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.430+0000 7ff6d4c00640 1 -- 172.21.15.28:0/648969101 wait complete. 2024-03-21T08:41:19.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 Processor -- start 2024-03-21T08:41:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 -- start start 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 0x7f08b00cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b015bf90 con 0x7f08b00b4800 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b015b1d0 con 0x7f08b00b4420 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08b5800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b015b4b0 con 0x7f08b0072190 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08a6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08b00b4800 0x7f08b00ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51006/0 (socket says 172.21.15.28:51006) 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08a6000640 1 -- 172.21.15.28:0/2659983148 learned_addr learned my addr 172.21.15.28:0/2659983148 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08af400640 1 --2- 172.21.15.28:0/2659983148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 0x7f08b00cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856984712 0 0) 0x7f08b015bf90 con 0x7f08b00b4800 2024-03-21T08:41:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0894003530 con 0x7f08b00b4800 2024-03-21T08:41:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713611702 0 0) 0x7f08b015b1d0 con 0x7f08b00b4420 2024-03-21T08:41:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b015bf90 con 0x7f08b00b4420 2024-03-21T08:41:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08af400640 1 -- 172.21.15.28:0/2659983148 >> v1:172.21.15.151:6789/0 conn(0x7f08b00b4420 legacy=0x7f08b0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08af400640 1 -- 172.21.15.28:0/2659983148 >> v1:172.21.15.28:6789/0 conn(0x7f08b00b4800 legacy=0x7f08b00ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.594+0000 7f08af400640 1 -- 172.21.15.28:0/2659983148 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b015b790 con 0x7f08b0072190 2024-03-21T08:41:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08af400640 1 --2- 172.21.15.28:0/2659983148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 0x7f08b00cd300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097f0 tx=0x7f08a801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ace4372042300dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a80b0070 con 0x7f08b0072190 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a8004230 con 0x7f08b0072190 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08ae000640 1 -- 172.21.15.28:0/2659983148 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8003620 con 0x7f08b0072190 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 -- 172.21.15.28:0/2659983148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 msgr2=0x7f08b00cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 --2- 172.21.15.28:0/2659983148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 0x7f08b00cd300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f08a80097f0 tx=0x7f08a801c4b0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 -- 172.21.15.28:0/2659983148 shutdown_connections 2024-03-21T08:41:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 --2- 172.21.15.28:0/2659983148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b0072190 0x7f08b00cd300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 -- 172.21.15.28:0/2659983148 >> 172.21.15.28:0/2659983148 conn(0x7f08b00674f0 msgr2=0x7f08b006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 -- 172.21.15.28:0/2659983148 shutdown_connections 2024-03-21T08:41:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.598+0000 7f08b5800640 1 -- 172.21.15.28:0/2659983148 wait complete. 2024-03-21T08:41:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 Processor -- start 2024-03-21T08:41:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- start start 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08b00b4420 0x7f08b014e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b014e930 0x7f08b0149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b006e430 con 0x7f08b0072190 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b006e2b0 con 0x7f08b00b4420 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b006e5b0 con 0x7f08b014e930 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53320/0 (socket says 172.21.15.28:53320) 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 -- 172.21.15.28:0/2815528860 learned_addr learned my addr 172.21.15.28:0/2815528860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08af400640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b014e930 0x7f08b0149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08a6000640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08b00b4420 0x7f08b014e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 -- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b014e930 msgr2=0x7f08b0149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b014e930 0x7f08b0149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 -- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08b00b4420 msgr2=0x7f08b014e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08b00b4420 0x7f08b014e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 -- 172.21.15.28:0/2815528860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b014f6a0 con 0x7f08b0072190 2024-03-21T08:41:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08aea00640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c800 tx=0x7f08a000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a0003b80 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a0003d40 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a000aa90 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b014f980 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.602+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b014ff40 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b014ed40 con 0x7f08b0072190 2024-03-21T08:41:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f08a000ac30 con 0x7f08b0072190 2024-03-21T08:41:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08a7400640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 0x7f088c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f08a0086170 con 0x7f08b0072190 2024-03-21T08:41:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08a6000640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 0x7f088c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.606+0000 7f08a6000640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 0x7f088c069fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f08b01f9b30 tx=0x7f0898007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.610+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f08b014ed40 con 0x7f08b0072190 2024-03-21T08:41:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08b0073430 con 0x7f08b0072190 2024-03-21T08:41:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08a7400640 1 -- 172.21.15.28:0/2815528860 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08a0056160 con 0x7f08b0072190 2024-03-21T08:41:19.776 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 msgr2=0x7f088c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 0x7f088c069fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f08b01f9b30 tx=0x7f0898007d40 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 msgr2=0x7f08b0142ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c800 tx=0x7f08a000ccc0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 shutdown_connections 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f088c067b20 0x7f088c069fe0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f08b014e930 0x7f08b0149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f08b00b4420 0x7f08b014e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 --2- 172.21.15.28:0/2815528860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08b0072190 0x7f08b0142ec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 >> 172.21.15.28:0/2815528860 conn(0x7f08b00674f0 msgr2=0x7f08b006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 shutdown_connections 2024-03-21T08:41:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.774+0000 7f08b5800640 1 -- 172.21.15.28:0/2815528860 wait complete. 2024-03-21T08:41:19.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 Processor -- start 2024-03-21T08:41:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 -- start start 2024-03-21T08:41:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c15e020 0x7f210c0cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f210c06b880 0x7f210c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c0d0760 con 0x7f210c0ccb70 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c0ce7b0 con 0x7f210c06b880 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2113a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0cfc90 con 0x7f210c15e020 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c15e020 0x7f210c0cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f210c06b880 0x7f210c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f210be00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f210c0ccb70 0x7f210c06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:44770/0 (socket says 172.21.15.28:44770) 2024-03-21T08:41:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f210be00640 1 -- 172.21.15.28:0/1335863338 learned_addr learned my addr 172.21.15.28:0/1335863338 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111a00640 1 -- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c15e020 msgr2=0x7f210c0cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111a00640 1 --2- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c15e020 0x7f210c0cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111a00640 1 -- 172.21.15.28:0/1335863338 >> v1:172.21.15.151:6789/0 conn(0x7f210c0ccb70 legacy=0x7f210c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f2111a00640 1 -- 172.21.15.28:0/1335863338 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0cfec0 con 0x7f210c06b880 2024-03-21T08:41:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.938+0000 7f210b400640 1 -- 172.21.15.28:0/1335863338 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710194577 0 0) 0x7f210c0d0760 con 0x7f210c0ccb70 2024-03-21T08:41:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2111a00640 1 --2- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c06b880 0x7f210c0ccfd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2104009d90 tx=0x7f210401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c4cd83e80e0e96f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f210b400640 1 -- 172.21.15.28:0/1335863338 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040b2070 con 0x7f210c06b880 2024-03-21T08:41:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f210b400640 1 -- 172.21.15.28:0/1335863338 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21040a5920 con 0x7f210c06b880 2024-03-21T08:41:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f210b400640 1 -- 172.21.15.28:0/1335863338 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040a5cf0 con 0x7f210c06b880 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 -- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c06b880 msgr2=0x7f210c0ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 --2- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c06b880 0x7f210c0ccfd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2104009d90 tx=0x7f210401c970 comp rx=0 tx=0).stop 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 -- 172.21.15.28:0/1335863338 shutdown_connections 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 --2- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c06b880 0x7f210c0ccfd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 --2- 172.21.15.28:0/1335863338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c15e020 0x7f210c0cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 -- 172.21.15.28:0/1335863338 >> 172.21.15.28:0/1335863338 conn(0x7f210c099a30 msgr2=0x7f210c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 -- 172.21.15.28:0/1335863338 shutdown_connections 2024-03-21T08:41:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.942+0000 7f2113a00640 1 -- 172.21.15.28:0/1335863338 wait complete. 2024-03-21T08:41:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 Processor -- start 2024-03-21T08:41:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 -- start start 2024-03-21T08:41:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 0x7f210c153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c15f410 con 0x7f210c0ccb70 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0cb590 con 0x7f210c15e020 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c15f590 con 0x7f210c06b880 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f210be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54922/0 (socket says 172.21.15.28:54922) 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f210be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53326/0 (socket says 172.21.15.28:53326) 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 0x7f210c153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 -- 172.21.15.28:0/825472624 learned_addr learned my addr 172.21.15.28:0/825472624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 -- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 msgr2=0x7f210c153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 0x7f210c153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 -- 172.21.15.28:0/825472624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 msgr2=0x7f210c152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 -- 172.21.15.28:0/825472624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c1589a0 con 0x7f210c06b880 2024-03-21T08:41:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 0x7f210c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f210be00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2111000640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f210000e960 tx=0x7f210000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100017070 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210000ca00 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c157320 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.946+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210000cdb0 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c1578e0 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c09eb60 con 0x7f210c06b880 2024-03-21T08:41:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f210001a070 con 0x7f210c06b880 2024-03-21T08:41:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f2108c00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 0x7f20ec069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f21000862a0 con 0x7f210c06b880 2024-03-21T08:41:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f210be00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 0x7f20ec069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.950+0000 7f210be00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 0x7f20ec069fe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f210c15d6c0 tx=0x7f20f8007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:19.962+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f210c1589a0 con 0x7f210c06b880 2024-03-21T08:41:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.130+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f210c04f4f0 con 0x7f210c06b880 2024-03-21T08:41:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.130+0000 7f2108c00640 1 -- 172.21.15.28:0/825472624 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2100056280 con 0x7f210c06b880 2024-03-21T08:41:20.134 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 msgr2=0x7f20ec069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 0x7f20ec069fe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f210c15d6c0 tx=0x7f20f8007d60 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 msgr2=0x7f210c06f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f210000e960 tx=0x7f210000ee20 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 shutdown_connections 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f210c15e020 0x7f210c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f210c0ccb70 0x7f210c152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f20ec067b20 0x7f20ec069fe0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 --2- 172.21.15.28:0/825472624 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f210c06b880 0x7f210c06f3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 >> 172.21.15.28:0/825472624 conn(0x7f210c099a30 msgr2=0x7f210c15b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 shutdown_connections 2024-03-21T08:41:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.134+0000 7f2113a00640 1 -- 172.21.15.28:0/825472624 wait complete. 2024-03-21T08:41:20.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.294+0000 7f225a200640 1 Processor -- start 2024-03-21T08:41:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.294+0000 7f225a200640 1 -- start start 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f225a200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f22540cea30 0x7f22540cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f225a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225415be30 con 0x7f225415ba50 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f225a200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540cc8e0 con 0x7f22540b0da0 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f225a200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540cc6b0 con 0x7f22540cea30 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2258e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f225415ba50 0x7f2254071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51008/0 (socket says 172.21.15.28:51008) 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2258e00640 1 -- 172.21.15.28:0/855755070 learned_addr learned my addr 172.21.15.28:0/855755070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2259800640 1 --2- 172.21.15.28:0/855755070 >> v2:172.21.15.151:3300/0 conn(0x7f22540cea30 0x7f22540cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2253400640 1 -- 172.21.15.28:0/855755070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875670912 0 0) 0x7f225415be30 con 0x7f225415ba50 2024-03-21T08:41:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2259800640 1 -- 172.21.15.28:0/855755070 >> v1:172.21.15.151:6789/0 conn(0x7f22540b0da0 legacy=0x7f22540ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2259800640 1 -- 172.21.15.28:0/855755070 >> v1:172.21.15.28:6789/0 conn(0x7f225415ba50 legacy=0x7f2254071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.298+0000 7f2259800640 1 -- 172.21.15.28:0/855755070 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540ccbc0 con 0x7f22540cea30 2024-03-21T08:41:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f2259800640 1 --2- 172.21.15.28:0/855755070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540cc170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f224c009d90 tx=0x7f224c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc6fd1f430e14eb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f2253400640 1 -- 172.21.15.28:0/855755070 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0b2070 con 0x7f22540cea30 2024-03-21T08:41:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f2253400640 1 -- 172.21.15.28:0/855755070 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c0a5940 con 0x7f22540cea30 2024-03-21T08:41:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f2253400640 1 -- 172.21.15.28:0/855755070 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0a5d10 con 0x7f22540cea30 2024-03-21T08:41:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- 172.21.15.28:0/855755070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 msgr2=0x7f22540cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 --2- 172.21.15.28:0/855755070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540cc170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f224c009d90 tx=0x7f224c01c970 comp rx=0 tx=0).stop 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- 172.21.15.28:0/855755070 shutdown_connections 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 --2- 172.21.15.28:0/855755070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540cc170 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- 172.21.15.28:0/855755070 >> 172.21.15.28:0/855755070 conn(0x7f225406fa90 msgr2=0x7f225406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- 172.21.15.28:0/855755070 shutdown_connections 2024-03-21T08:41:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- 172.21.15.28:0/855755070 wait complete. 2024-03-21T08:41:20.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 Processor -- start 2024-03-21T08:41:20.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- start start 2024-03-21T08:41:20.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f225415ba50 0x7f22540b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540b11b0 con 0x7f22540b0da0 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540b53f0 con 0x7f22540cea30 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.302+0000 7f225a200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22541695d0 con 0x7f225415ba50 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53330/0 (socket says 172.21.15.28:53330) 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 -- 172.21.15.28:0/1470849958 learned_addr learned my addr 172.21.15.28:0/1470849958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2259800640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f225415ba50 0x7f22540b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2253e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 -- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f225415ba50 msgr2=0x7f22540b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f225415ba50 0x7f22540b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 -- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 msgr2=0x7f22540c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 -- 172.21.15.28:0/1470849958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540c7d30 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2258e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f224800c4d0 tx=0x7f224800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248018070 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22480074e0 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2248004a30 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2254163910 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2254163ed0 con 0x7f22540b0da0 2024-03-21T08:41:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225404f410 con 0x7f22540b0da0 2024-03-21T08:41:20.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.306+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2248004bd0 con 0x7f22540b0da0 2024-03-21T08:41:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.310+0000 7f2250c00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 0x7f223406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.310+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2248085410 con 0x7f22540b0da0 2024-03-21T08:41:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.310+0000 7f2253e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 0x7f223406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.310+0000 7f2253e00640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 0x7f223406a010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f22540cbbe0 tx=0x7f2240009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.314+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2248055490 con 0x7f22540b0da0 2024-03-21T08:41:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.478+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2254068320 con 0x7f22540b0da0 2024-03-21T08:41:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f2250c00640 1 -- 172.21.15.28:0/1470849958 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2248059530 con 0x7f22540b0da0 2024-03-21T08:41:20.484 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 msgr2=0x7f223406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 0x7f223406a010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f22540cbbe0 tx=0x7f2240009040 comp rx=0 tx=0).stop 2024-03-21T08:41:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 msgr2=0x7f2254166360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f224800c4d0 tx=0x7f224800c990 comp rx=0 tx=0).stop 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 shutdown_connections 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2234067b50 0x7f223406a010 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f225415ba50 0x7f22540b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f22540cea30 0x7f22540c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 --2- 172.21.15.28:0/1470849958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22540b0da0 0x7f2254166360 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 >> 172.21.15.28:0/1470849958 conn(0x7f225406fa90 msgr2=0x7f22540cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 shutdown_connections 2024-03-21T08:41:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.482+0000 7f225a200640 1 -- 172.21.15.28:0/1470849958 wait complete. 2024-03-21T08:41:20.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:41:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.642+0000 7ff4ff200640 1 Processor -- start 2024-03-21T08:41:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.642+0000 7ff4ff200640 1 -- start start 2024-03-21T08:41:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.642+0000 7ff4ff200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 0x7ff4f809aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.642+0000 7ff4ff200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f809a1d0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4ff200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f809a4b0 con 0x7ff4f815be50 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4ff200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f815c230 con 0x7ff4f80c6f50 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4f8099bf0 0x7ff4f809a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36698/0 (socket says 172.21.15.28:36698) 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4efe00640 1 -- 172.21.15.28:0/1101824159 learned_addr learned my addr 172.21.15.28:0/1101824159 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4fd200640 1 --2- 172.21.15.28:0/1101824159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 0x7ff4f809aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531151464 0 0) 0x7ff4f809a1d0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc003530 con 0x7ff4f8099bf0 2024-03-21T08:41:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83149069 0 0) 0x7ff4f809a4b0 con 0x7ff4f815be50 2024-03-21T08:41:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f809a1d0 con 0x7ff4f815be50 2024-03-21T08:41:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4fd200640 1 -- 172.21.15.28:0/1101824159 >> v1:172.21.15.151:6789/0 conn(0x7ff4f815be50 legacy=0x7ff4f80d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4fd200640 1 -- 172.21.15.28:0/1101824159 >> v1:172.21.15.28:6789/0 conn(0x7ff4f8099bf0 legacy=0x7ff4f809a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4fd200640 1 -- 172.21.15.28:0/1101824159 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f80c5740 con 0x7ff4f80c6f50 2024-03-21T08:41:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4fd200640 1 --2- 172.21.15.28:0/1101824159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 0x7ff4f809aea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009dc0 tx=0x7ff4f001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ff245de15d56d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.646+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00b2070 con 0x7ff4f80c6f50 2024-03-21T08:41:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f00a57c0 con 0x7ff4f80c6f50 2024-03-21T08:41:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7400640 1 -- 172.21.15.28:0/1101824159 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00a5b70 con 0x7ff4f80c6f50 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- 172.21.15.28:0/1101824159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 msgr2=0x7ff4f809aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 --2- 172.21.15.28:0/1101824159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 0x7ff4f809aea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009dc0 tx=0x7ff4f001c900 comp rx=0 tx=0).stop 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- 172.21.15.28:0/1101824159 shutdown_connections 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 --2- 172.21.15.28:0/1101824159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f80c6f50 0x7ff4f809aea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- 172.21.15.28:0/1101824159 >> 172.21.15.28:0/1101824159 conn(0x7ff4f80b3540 msgr2=0x7ff4f80b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- 172.21.15.28:0/1101824159 shutdown_connections 2024-03-21T08:41:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- 172.21.15.28:0/1101824159 wait complete. 2024-03-21T08:41:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 Processor -- start 2024-03-21T08:41:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- start start 2024-03-21T08:41:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f80c6f50 0x7ff4f806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f815be50 0x7ff4f80c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f81698c0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f8169740 con 0x7ff4f80c6f50 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4ff200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f8169a40 con 0x7ff4f815be50 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51166/0 (socket says 172.21.15.28:51166) 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 -- 172.21.15.28:0/230668315 learned_addr learned my addr 172.21.15.28:0/230668315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4efe00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f80c6f50 0x7ff4f806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4fd200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f815be50 0x7ff4f80c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 -- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f815be50 msgr2=0x7ff4f80c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f815be50 0x7ff4f80c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 -- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f80c6f50 msgr2=0x7ff4f806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.650+0000 7ff4f7e00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f80c6f50 0x7ff4f806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f7e00640 1 -- 172.21.15.28:0/230668315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8070920 con 0x7ff4f8099bf0 2024-03-21T08:41:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f7e00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c460 tx=0x7ff4e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8017070 con 0x7ff4f8099bf0 2024-03-21T08:41:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e80052f0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8003840 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f806c490 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f806ca50 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f809f9c0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff4e801a070 con 0x7ff4f8099bf0 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 0x7ff4d4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff4e8085620 con 0x7ff4f8099bf0 2024-03-21T08:41:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.654+0000 7ff4efe00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 0x7ff4d4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.658+0000 7ff4efe00640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 0x7ff4d4069fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4f806e3e0 tx=0x7ff4e0007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.658+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff4e8055600 con 0x7ff4f8099bf0 2024-03-21T08:41:20.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.834+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff4f80685b0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.834+0000 7ff4f4c00640 1 -- 172.21.15.28:0/230668315 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff4e80596a0 con 0x7ff4f8099bf0 2024-03-21T08:41:20.838 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 msgr2=0x7ff4d4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 0x7ff4d4069fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4f806e3e0 tx=0x7ff4e0007d40 comp rx=0 tx=0).stop 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 msgr2=0x7ff4f814ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c460 tx=0x7ff4e800c920 comp rx=0 tx=0).stop 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 shutdown_connections 2024-03-21T08:41:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4d4067b20 0x7ff4d4069fe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4f815be50 0x7ff4f80c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4f80c6f50 0x7ff4f806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 --2- 172.21.15.28:0/230668315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4f8099bf0 0x7ff4f814ae00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 >> 172.21.15.28:0/230668315 conn(0x7ff4f80b3540 msgr2=0x7ff4f80c7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 shutdown_connections 2024-03-21T08:41:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:20.838+0000 7ff4ff200640 1 -- 172.21.15.28:0/230668315 wait complete. 2024-03-21T08:41:20.859 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:41:20.859 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:41:20.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 Processor -- start 2024-03-21T08:41:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 -- start start 2024-03-21T08:41:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2a815e0c0 0x7fb2a806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a809a010 0x7fb2a80d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a80cc1d0 con 0x7fb2a8099b80 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a80d1730 con 0x7fb2a815e0c0 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80d1200 con 0x7fb2a809a010 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2a815e0c0 0x7fb2a806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2a8099b80 0x7fb2a80ccc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36708/0 (socket says 172.21.15.28:36708) 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7400640 1 -- 172.21.15.28:0/183442924 learned_addr learned my addr 172.21.15.28:0/183442924 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2aca00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a809a010 0x7fb2a80d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7e00640 1 -- 172.21.15.28:0/183442924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a809a010 msgr2=0x7fb2a80d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7e00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a809a010 0x7fb2a80d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7e00640 1 -- 172.21.15.28:0/183442924 >> v1:172.21.15.28:6789/0 conn(0x7fb2a8099b80 legacy=0x7fb2a80ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.050+0000 7fb2a7e00640 1 -- 172.21.15.28:0/183442924 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a806d450 con 0x7fb2a815e0c0 2024-03-21T08:41:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2a7e00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a815e0c0 0x7fb2a806cf10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80d1380 tx=0x7fb29801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152214e18c89c503 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2a6a00640 1 -- 172.21.15.28:0/183442924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2980aa600 con 0x7fb2a815e0c0 2024-03-21T08:41:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2a6a00640 1 -- 172.21.15.28:0/183442924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2980af070 con 0x7fb2a815e0c0 2024-03-21T08:41:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2a6a00640 1 -- 172.21.15.28:0/183442924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2980aa9c0 con 0x7fb2a815e0c0 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 -- 172.21.15.28:0/183442924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a815e0c0 msgr2=0x7fb2a806cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a815e0c0 0x7fb2a806cf10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80d1380 tx=0x7fb29801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 -- 172.21.15.28:0/183442924 shutdown_connections 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a809a010 0x7fb2a80d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 --2- 172.21.15.28:0/183442924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a815e0c0 0x7fb2a806cf10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 -- 172.21.15.28:0/183442924 >> 172.21.15.28:0/183442924 conn(0x7fb2a80b34d0 msgr2=0x7fb2a80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 -- 172.21.15.28:0/183442924 shutdown_connections 2024-03-21T08:41:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.054+0000 7fb2aea00640 1 -- 172.21.15.28:0/183442924 wait complete. 2024-03-21T08:41:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 Processor -- start 2024-03-21T08:41:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- start start 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8099b80 0x7fb2a81549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a815e0c0 0x7fb2a80b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a815e4a0 con 0x7fb2a809a010 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80cc750 con 0x7fb2a8099b80 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a80cdb90 con 0x7fb2a815e0c0 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51184/0 (socket says 172.21.15.28:51184) 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 -- 172.21.15.28:0/2490623752 learned_addr learned my addr 172.21.15.28:0/2490623752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7e00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8099b80 0x7fb2a81549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aca00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a815e0c0 0x7fb2a80b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 -- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a815e0c0 msgr2=0x7fb2a80b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a815e0c0 0x7fb2a80b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 -- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8099b80 msgr2=0x7fb2a81549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8099b80 0x7fb2a81549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 -- 172.21.15.28:0/2490623752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8071200 con 0x7fb2a809a010 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2a7400640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c420 tx=0x7fb29400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294017070 con 0x7fb2a809a010 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294005cf0 con 0x7fb2a809a010 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940072f0 con 0x7fb2a809a010 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a8154f20 con 0x7fb2a809a010 2024-03-21T08:41:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a81554a0 con 0x7fb2a809a010 2024-03-21T08:41:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.058+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a804f410 con 0x7fb2a809a010 2024-03-21T08:41:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.062+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb29401a070 con 0x7fb2a809a010 2024-03-21T08:41:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.062+0000 7fb29fe00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 0x7fb28806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.062+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb2940855f0 con 0x7fb2a809a010 2024-03-21T08:41:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.062+0000 7fb2a7e00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 0x7fb28806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.062+0000 7fb2a7e00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 0x7fb28806a010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2980ae000 tx=0x7fb2980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.066+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb2940555d0 con 0x7fb2a809a010 2024-03-21T08:41:21.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.230+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a815cd30 con 0x7fb2a809a010 2024-03-21T08:41:21.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.230+0000 7fb29fe00640 1 -- 172.21.15.28:0/2490623752 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7fb294059670 con 0x7fb2a809a010 2024-03-21T08:41:21.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:21.234 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:21.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 msgr2=0x7fb28806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 0x7fb28806a010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2980ae000 tx=0x7fb2980a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 msgr2=0x7fb2a80b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c420 tx=0x7fb29400c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 shutdown_connections 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb288067b50 0x7fb28806a010 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a815e0c0 0x7fb2a80b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a809a010 0x7fb2a80b4540 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 --2- 172.21.15.28:0/2490623752 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8099b80 0x7fb2a81549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 >> 172.21.15.28:0/2490623752 conn(0x7fb2a80b34d0 msgr2=0x7fb2a80b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 shutdown_connections 2024-03-21T08:41:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.234+0000 7fb2aea00640 1 -- 172.21.15.28:0/2490623752 wait complete. 2024-03-21T08:41:21.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:21.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:21.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.394+0000 7fbf75a00640 1 Processor -- start 2024-03-21T08:41:21.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.394+0000 7fbf75a00640 1 -- start start 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf7006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbf700cde40 0x7fbf70071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7015cbb0 0x7fbf700cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7015c720 con 0x7fbf7015cbb0 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf700723a0 con 0x7fbf700cde40 2024-03-21T08:41:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf75a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700ce5b0 con 0x7fbf70071a80 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7015cbb0 0x7fbf700cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7015cbb0 0x7fbf700cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51194/0 (socket says 172.21.15.28:51194) 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 -- 172.21.15.28:0/176283776 learned_addr learned my addr 172.21.15.28:0/176283776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6ea00640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf7006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf66000640 1 --2- 172.21.15.28:0/176283776 >> v2:172.21.15.151:3300/0 conn(0x7fbf700cde40 0x7fbf70071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 -- 172.21.15.28:0/176283776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 msgr2=0x7fbf7006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf7006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 -- 172.21.15.28:0/176283776 >> v2:172.21.15.151:3300/0 conn(0x7fbf700cde40 msgr2=0x7fbf70071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 --2- 172.21.15.28:0/176283776 >> v2:172.21.15.151:3300/0 conn(0x7fbf700cde40 0x7fbf70071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.398+0000 7fbf6f400640 1 -- 172.21.15.28:0/176283776 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700ce890 con 0x7fbf7015cbb0 2024-03-21T08:41:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf6f400640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7015cbb0 0x7fbf700cc4b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b4f60 tx=0x7fbf6801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dd3b9a91a857 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf6e000640 1 -- 172.21.15.28:0/176283776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf680ab640 con 0x7fbf7015cbb0 2024-03-21T08:41:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf6e000640 1 -- 172.21.15.28:0/176283776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf680b0070 con 0x7fbf7015cbb0 2024-03-21T08:41:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf6e000640 1 -- 172.21.15.28:0/176283776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf680aba00 con 0x7fbf7015cbb0 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- 172.21.15.28:0/176283776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7015cbb0 msgr2=0x7fbf700cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7015cbb0 0x7fbf700cc4b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf700b4f60 tx=0x7fbf6801c460 comp rx=0 tx=0).stop 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- 172.21.15.28:0/176283776 shutdown_connections 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7015cbb0 0x7fbf700cc4b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- 172.21.15.28:0/176283776 >> v2:172.21.15.151:3300/0 conn(0x7fbf700cde40 0x7fbf70071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- 172.21.15.28:0/176283776 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf7006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- 172.21.15.28:0/176283776 >> 172.21.15.28:0/176283776 conn(0x7fbf700674f0 msgr2=0x7fbf7006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- 172.21.15.28:0/176283776 shutdown_connections 2024-03-21T08:41:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- 172.21.15.28:0/176283776 wait complete. 2024-03-21T08:41:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 Processor -- start 2024-03-21T08:41:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- start start 2024-03-21T08:41:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf70137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf7015cbb0 0x7fbf7015aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700ce220 con 0x7fbf700cde40 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70144b20 con 0x7fbf7015cbb0 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.402+0000 7fbf75a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7006e2b0 con 0x7fbf70071a80 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51202/0 (socket says 172.21.15.28:51202) 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf6ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf70137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 -- 172.21.15.28:0/3926899622 learned_addr learned my addr 172.21.15.28:0/3926899622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf6f400640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf7015cbb0 0x7fbf7015aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 -- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 msgr2=0x7fbf70137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf70137f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 -- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf7015cbb0 msgr2=0x7fbf7015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf7015cbb0 0x7fbf7015aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 -- 172.21.15.28:0/3926899622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70138560 con 0x7fbf700cde40 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf66000640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e960 tx=0x7fbf5800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58017070 con 0x7fbf700cde40 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800ca00 con 0x7fbf700cde40 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf70138840 con 0x7fbf700cde40 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800cdb0 con 0x7fbf700cde40 2024-03-21T08:41:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf70138e00 con 0x7fbf700cde40 2024-03-21T08:41:21.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf70073430 con 0x7fbf700cde40 2024-03-21T08:41:21.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.406+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbf5801a070 con 0x7fbf700cde40 2024-03-21T08:41:21.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.410+0000 7fbf67400640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 0x7fbf4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.410+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf580865a0 con 0x7fbf700cde40 2024-03-21T08:41:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.410+0000 7fbf6ea00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 0x7fbf4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.410+0000 7fbf6ea00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 0x7fbf4c069fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf700cf620 tx=0x7fbf6000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.414+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbf58056580 con 0x7fbf700cde40 2024-03-21T08:41:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.578+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf7004f4f0 con 0x7fbf700cde40 2024-03-21T08:41:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.578+0000 7fbf67400640 1 -- 172.21.15.28:0/3926899622 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf5805a620 con 0x7fbf700cde40 2024-03-21T08:41:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 msgr2=0x7fbf4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 0x7fbf4c069fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf700cf620 tx=0x7fbf6000b040 comp rx=0 tx=0).stop 2024-03-21T08:41:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 msgr2=0x7fbf70159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e960 tx=0x7fbf5800ee20 comp rx=0 tx=0).stop 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 shutdown_connections 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf7015cbb0 0x7fbf7015aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf700cde40 0x7fbf70159940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf4c067b20 0x7fbf4c069fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 --2- 172.21.15.28:0/3926899622 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf70071a80 0x7fbf70137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 >> 172.21.15.28:0/3926899622 conn(0x7fbf700674f0 msgr2=0x7fbf7015b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 shutdown_connections 2024-03-21T08:41:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.582+0000 7fbf75a00640 1 -- 172.21.15.28:0/3926899622 wait complete. 2024-03-21T08:41:21.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:21.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 Processor -- start 2024-03-21T08:41:21.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 -- start start 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 0x7f24d00cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d00d0760 con 0x7f24d00ccb70 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d00cfec0 con 0x7f24d006b880 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00ce7b0 con 0x7f24d015e020 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24d00ccb70 0x7f24d006b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36718/0 (socket says 172.21.15.28:36718) 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24c7e00640 1 -- 172.21.15.28:0/4042344073 learned_addr learned my addr 172.21.15.28:0/4042344073 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d4e00640 1 --2- 172.21.15.28:0/4042344073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 0x7f24d00cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188915678 0 0) 0x7f24d00d0760 con 0x7f24d00ccb70 2024-03-21T08:41:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b4003530 con 0x7f24d00ccb70 2024-03-21T08:41:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242854557 0 0) 0x7f24d00cfec0 con 0x7f24d006b880 2024-03-21T08:41:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d00d0760 con 0x7f24d006b880 2024-03-21T08:41:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d4e00640 1 -- 172.21.15.28:0/4042344073 >> v1:172.21.15.151:6789/0 conn(0x7f24d006b880 legacy=0x7f24d015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d4e00640 1 -- 172.21.15.28:0/4042344073 >> v1:172.21.15.28:6789/0 conn(0x7f24d00ccb70 legacy=0x7f24d006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.746+0000 7f24d4e00640 1 -- 172.21.15.28:0/4042344073 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d00d01a0 con 0x7f24d015e020 2024-03-21T08:41:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d4e00640 1 --2- 172.21.15.28:0/4042344073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 0x7f24d00cb050 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097f0 tx=0x7f24c001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fd80bb24272289a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00b0070 con 0x7f24d015e020 2024-03-21T08:41:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00ab600 con 0x7f24d015e020 2024-03-21T08:41:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24cfe00640 1 -- 172.21.15.28:0/4042344073 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00ab9b0 con 0x7f24d015e020 2024-03-21T08:41:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 -- 172.21.15.28:0/4042344073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 msgr2=0x7f24d00cb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 --2- 172.21.15.28:0/4042344073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 0x7f24d00cb050 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097f0 tx=0x7f24c001c4e0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 -- 172.21.15.28:0/4042344073 shutdown_connections 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 --2- 172.21.15.28:0/4042344073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d015e020 0x7f24d00cb050 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 -- 172.21.15.28:0/4042344073 >> 172.21.15.28:0/4042344073 conn(0x7f24d0099a30 msgr2=0x7f24d0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 -- 172.21.15.28:0/4042344073 shutdown_connections 2024-03-21T08:41:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.750+0000 7f24d7800640 1 -- 172.21.15.28:0/4042344073 wait complete. 2024-03-21T08:41:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 Processor -- start 2024-03-21T08:41:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- start start 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 0x7f24d0151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24d015e020 0x7f24d0149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00cb590 con 0x7f24d00ccb70 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d015e790 con 0x7f24d015e020 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d016c340 con 0x7f24d006b880 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51204/0 (socket says 172.21.15.28:51204) 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24c7e00640 1 -- 172.21.15.28:0/790849688 learned_addr learned my addr 172.21.15.28:0/790849688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 0x7f24d0151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d5800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24d015e020 0x7f24d0149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 -- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24d015e020 msgr2=0x7f24d0149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24d015e020 0x7f24d0149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 -- 172.21.15.28:0/790849688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 msgr2=0x7f24d0148020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 -- 172.21.15.28:0/790849688 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d0152830 con 0x7f24d006b880 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24c7e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d4e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 0x7f24d0151c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24d00727a0 tx=0x7f24c0019280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00bb070 con 0x7f24d006b880 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c00b7460 con 0x7f24d006b880 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d014a520 con 0x7f24d006b880 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00b0040 con 0x7f24d006b880 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.754+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d0152550 con 0x7f24d006b880 2024-03-21T08:41:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d0149c10 con 0x7f24d006b880 2024-03-21T08:41:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f24c00be070 con 0x7f24d006b880 2024-03-21T08:41:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24cd600640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 0x7f24ac069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:21.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f24c01207b0 con 0x7f24d006b880 2024-03-21T08:41:21.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24c7e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 0x7f24ac069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:21.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.758+0000 7f24c7e00640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 0x7f24ac069fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f24d0157070 tx=0x7f24b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.762+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f24c00160d0 con 0x7f24d006b880 2024-03-21T08:41:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.926+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24d004f4f0 con 0x7f24d006b880 2024-03-21T08:41:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.926+0000 7f24cd600640 1 -- 172.21.15.28:0/790849688 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f24c00f0940 con 0x7f24d006b880 2024-03-21T08:41:21.931 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 msgr2=0x7f24ac069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 0x7f24ac069fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f24d0157070 tx=0x7f24b8009040 comp rx=0 tx=0).stop 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 msgr2=0x7f24d0151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 0x7f24d0151c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24d00727a0 tx=0x7f24c0019280 comp rx=0 tx=0).stop 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 shutdown_connections 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24d015e020 0x7f24d0149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24d00ccb70 0x7f24d0148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24ac067b20 0x7f24ac069fe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 --2- 172.21.15.28:0/790849688 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24d006b880 0x7f24d0151c90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 >> 172.21.15.28:0/790849688 conn(0x7f24d0099a30 msgr2=0x7f24d00cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 shutdown_connections 2024-03-21T08:41:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:21.930+0000 7f24d7800640 1 -- 172.21.15.28:0/790849688 wait complete. 2024-03-21T08:41:21.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.094+0000 7f297f600640 1 Processor -- start 2024-03-21T08:41:22.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.094+0000 7f297f600640 1 -- start start 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297f600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297f600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29780b48a0 con 0x7f297815df10 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297f600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29780d1620 con 0x7f29780d0e00 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297f600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29780cc0a0 con 0x7f297806d3a0 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297cc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f297815df10 0x7f297806cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36724/0 (socket says 172.21.15.28:36724) 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f297cc00640 1 -- 172.21.15.28:0/1756144479 learned_addr learned my addr 172.21.15.28:0/1756144479 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f296fe00640 1 --2- 172.21.15.28:0/1756144479 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405260051 0 0) 0x7f2968001120 con 0x7f297815df10 2024-03-21T08:41:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f295c003530 con 0x7f297815df10 2024-03-21T08:41:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885984508 0 0) 0x7f29780d1620 con 0x7f29780d0e00 2024-03-21T08:41:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2968001120 con 0x7f29780d0e00 2024-03-21T08:41:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f296fe00640 1 -- 172.21.15.28:0/1756144479 >> v1:172.21.15.151:6789/0 conn(0x7f29780d0e00 legacy=0x7f29780ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f296fe00640 1 -- 172.21.15.28:0/1756144479 >> v1:172.21.15.28:6789/0 conn(0x7f297815df10 legacy=0x7f297806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.098+0000 7f296fe00640 1 -- 172.21.15.28:0/1756144479 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780cc2f0 con 0x7f297806d3a0 2024-03-21T08:41:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f296fe00640 1 --2- 172.21.15.28:0/1756144479 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780d08c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f29600097f0 tx=0x7f296001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5de3c76532e1fabf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600b1070 con 0x7f297806d3a0 2024-03-21T08:41:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29600a4e30 con 0x7f297806d3a0 2024-03-21T08:41:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f2977e00640 1 -- 172.21.15.28:0/1756144479 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600ac640 con 0x7f297806d3a0 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- 172.21.15.28:0/1756144479 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 msgr2=0x7f29780d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 --2- 172.21.15.28:0/1756144479 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780d08c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f29600097f0 tx=0x7f296001c450 comp rx=0 tx=0).stop 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- 172.21.15.28:0/1756144479 shutdown_connections 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 --2- 172.21.15.28:0/1756144479 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780d08c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- 172.21.15.28:0/1756144479 >> 172.21.15.28:0/1756144479 conn(0x7f29780b3540 msgr2=0x7f29780b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- 172.21.15.28:0/1756144479 shutdown_connections 2024-03-21T08:41:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- 172.21.15.28:0/1756144479 wait complete. 2024-03-21T08:41:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 Processor -- start 2024-03-21T08:41:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- start start 2024-03-21T08:41:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f297815df10 0x7f297815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297816bd40 con 0x7f29780d0e00 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297816bbc0 con 0x7f297815df10 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.102+0000 7f297f600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297816bec0 con 0x7f297806d3a0 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51218/0 (socket says 172.21.15.28:51218) 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 -- 172.21.15.28:0/2094143403 learned_addr learned my addr 172.21.15.28:0/2094143403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f297cc00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f297d600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f297815df10 0x7f297815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 -- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 msgr2=0x7f29780c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 -- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f297815df10 msgr2=0x7f297815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f297815df10 0x7f297815fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 -- 172.21.15.28:0/2094143403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780c9410 con 0x7f29780d0e00 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f296fe00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29600097f0 tx=0x7f296001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600b1040 con 0x7f29780d0e00 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29600bc070 con 0x7f29780d0e00 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600acda0 con 0x7f29780d0e00 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29780c99a0 con 0x7f29780d0e00 2024-03-21T08:41:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29780c9f60 con 0x7f29780d0e00 2024-03-21T08:41:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297804f4f0 con 0x7f29780d0e00 2024-03-21T08:41:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.106+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f29600c1460 con 0x7f29780d0e00 2024-03-21T08:41:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.110+0000 7f2975600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 0x7f2954069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.110+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f296011ff90 con 0x7f29780d0e00 2024-03-21T08:41:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.110+0000 7f297cc00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 0x7f2954069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.110+0000 7f297cc00640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 0x7f2954069fe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2968007fd0 tx=0x7f2968007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.114+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f29600160d0 con 0x7f29780d0e00 2024-03-21T08:41:22.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2978070fb0 con 0x7f29780d0e00 2024-03-21T08:41:22.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f2975600640 1 -- 172.21.15.28:0/2094143403 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29600eff70 con 0x7f29780d0e00 2024-03-21T08:41:22.276 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 msgr2=0x7f2954069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 0x7f2954069fe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2968007fd0 tx=0x7f2968007d80 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 msgr2=0x7f29780caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29600097f0 tx=0x7f296001ac90 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 shutdown_connections 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f297815df10 0x7f297815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29780d0e00 0x7f29780caf00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2954067b20 0x7f2954069fe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 --2- 172.21.15.28:0/2094143403 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f297806d3a0 0x7f29780c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 >> 172.21.15.28:0/2094143403 conn(0x7f29780b3540 msgr2=0x7f297806fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 shutdown_connections 2024-03-21T08:41:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.274+0000 7f297f600640 1 -- 172.21.15.28:0/2094143403 wait complete. 2024-03-21T08:41:22.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:22.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 Processor -- start 2024-03-21T08:41:22.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 -- start start 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8006d3a0 0x7f7e800d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e800d0e00 0x7f7e800ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e800b48a0 con 0x7f7e8015df10 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e800cc0a0 con 0x7f7e800d0e00 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e85400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800cc940 con 0x7f7e8006d3a0 2024-03-21T08:41:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e7e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e8015df10 0x7f7e8006cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36736/0 (socket says 172.21.15.28:36736) 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e7e800640 1 -- 172.21.15.28:0/2436604935 learned_addr learned my addr 172.21.15.28:0/2436604935 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e7f200640 1 --2- 172.21.15.28:0/2436604935 >> v2:172.21.15.28:3300/0 conn(0x7f7e800d0e00 0x7f7e800ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e75e00640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8006d3a0 0x7f7e800d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e7de00640 1 -- 172.21.15.28:0/2436604935 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726134962 0 0) 0x7f7e70001120 con 0x7f7e8015df10 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.438+0000 7f7e7de00640 1 -- 172.21.15.28:0/2436604935 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e64003530 con 0x7f7e8015df10 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7f200640 1 -- 172.21.15.28:0/2436604935 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8006d3a0 msgr2=0x7f7e800d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7f200640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8006d3a0 0x7f7e800d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7f200640 1 -- 172.21.15.28:0/2436604935 >> v1:172.21.15.28:6789/0 conn(0x7f7e8015df10 legacy=0x7f7e8006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7f200640 1 -- 172.21.15.28:0/2436604935 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800d1620 con 0x7f7e800d0e00 2024-03-21T08:41:22.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7f200640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e800d0e00 0x7f7e800ccdd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097c0 tx=0x7f7e7801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873106722a7a892b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7de00640 1 -- 172.21.15.28:0/2436604935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780b0070 con 0x7f7e800d0e00 2024-03-21T08:41:22.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7de00640 1 -- 172.21.15.28:0/2436604935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e780ab600 con 0x7f7e800d0e00 2024-03-21T08:41:22.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e7de00640 1 -- 172.21.15.28:0/2436604935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e780ab9b0 con 0x7f7e800d0e00 2024-03-21T08:41:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 -- 172.21.15.28:0/2436604935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e800d0e00 msgr2=0x7f7e800ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e800d0e00 0x7f7e800ccdd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097c0 tx=0x7f7e7801c410 comp rx=0 tx=0).stop 2024-03-21T08:41:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 -- 172.21.15.28:0/2436604935 shutdown_connections 2024-03-21T08:41:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e800d0e00 0x7f7e800ccdd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 --2- 172.21.15.28:0/2436604935 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8006d3a0 0x7f7e800d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 -- 172.21.15.28:0/2436604935 >> 172.21.15.28:0/2436604935 conn(0x7f7e800b3540 msgr2=0x7f7e800b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 -- 172.21.15.28:0/2436604935 shutdown_connections 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.442+0000 7f7e85400640 1 -- 172.21.15.28:0/2436604935 wait complete. 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 Processor -- start 2024-03-21T08:41:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- start start 2024-03-21T08:41:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e800d0e00 0x7f7e8015e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 0x7f7e800ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800cdca0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800cdb20 con 0x7f7e800d0e00 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800cde20 con 0x7f7e8015df10 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51244/0 (socket says 172.21.15.28:51244) 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 -- 172.21.15.28:0/1467688561 learned_addr learned my addr 172.21.15.28:0/1467688561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7f200640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 0x7f7e800ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e75e00640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e800d0e00 0x7f7e8015e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 -- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 msgr2=0x7f7e800ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 0x7f7e800ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 -- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e800d0e00 msgr2=0x7f7e8015e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e800d0e00 0x7f7e8015e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 -- 172.21.15.28:0/1467688561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8015efd0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7f200640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 0x7f7e800ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e7e800640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c760 tx=0x7f7e7000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70017070 con 0x7f7e8006d3a0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e70005ae0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8015f2b0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70007e30 con 0x7f7e8006d3a0 2024-03-21T08:41:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.446+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8015f870 con 0x7f7e8006d3a0 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8004f4f0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7e7001a070 con 0x7f7e8006d3a0 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e77400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 0x7f7e5c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7e70085610 con 0x7f7e8006d3a0 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e75e00640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 0x7f7e5c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.450+0000 7f7e75e00640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 0x7f7e5c069fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e8009b040 tx=0x7f7e6800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.454+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7e800b48a0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.614+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7e800b6460 con 0x7f7e8006d3a0 2024-03-21T08:41:22.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e77400640 1 -- 172.21.15.28:0/1467688561 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7e700555f0 con 0x7f7e8006d3a0 2024-03-21T08:41:22.620 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 msgr2=0x7f7e5c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 0x7f7e5c069fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e8009b040 tx=0x7f7e6800b040 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 msgr2=0x7f7e8015afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c760 tx=0x7f7e7000cc20 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 shutdown_connections 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e5c067b20 0x7f7e5c069fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7e8015df10 0x7f7e800ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7e800d0e00 0x7f7e8015e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 --2- 172.21.15.28:0/1467688561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e8006d3a0 0x7f7e8015afd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 >> 172.21.15.28:0/1467688561 conn(0x7f7e800b3540 msgr2=0x7f7e80071510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 shutdown_connections 2024-03-21T08:41:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.618+0000 7f7e85400640 1 -- 172.21.15.28:0/1467688561 wait complete. 2024-03-21T08:41:22.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:41:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 Processor -- start 2024-03-21T08:41:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 -- start start 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6406f0a0 0x7fbc640cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc6415cbb0 0x7fbc640cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6406f500 con 0x7fbc64071a80 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc640ce380 con 0x7fbc6415cbb0 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6c200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6415c720 con 0x7fbc6406f0a0 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc6415cbb0 0x7fbc640cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc69800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fbc64071a80 0x7fbc64071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39584/0 (socket says 172.21.15.28:39584) 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc69800640 1 -- 172.21.15.28:0/4031006421 learned_addr learned my addr 172.21.15.28:0/4031006421 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc68e00640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6406f0a0 0x7fbc640cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6a200640 1 -- 172.21.15.28:0/4031006421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6406f0a0 msgr2=0x7fbc640cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6a200640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6406f0a0 0x7fbc640cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6a200640 1 -- 172.21.15.28:0/4031006421 >> v1:172.21.15.151:6789/0 conn(0x7fbc64071a80 legacy=0x7fbc64071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc6a200640 1 -- 172.21.15.28:0/4031006421 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc640ce5b0 con 0x7fbc6415cbb0 2024-03-21T08:41:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.782+0000 7fbc63e00640 1 -- 172.21.15.28:0/4031006421 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414034131 0 0) 0x7fbc6406f500 con 0x7fbc64071a80 2024-03-21T08:41:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6a200640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6415cbb0 0x7fbc640cc4b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009d90 tx=0x7fbc5c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40b3c322b0fb1e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc63e00640 1 -- 172.21.15.28:0/4031006421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0b2070 con 0x7fbc6415cbb0 2024-03-21T08:41:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc63e00640 1 -- 172.21.15.28:0/4031006421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c0a5970 con 0x7fbc6415cbb0 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc63e00640 1 -- 172.21.15.28:0/4031006421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0a5d40 con 0x7fbc6415cbb0 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- 172.21.15.28:0/4031006421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6415cbb0 msgr2=0x7fbc640cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6415cbb0 0x7fbc640cc4b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009d90 tx=0x7fbc5c01c8b0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- 172.21.15.28:0/4031006421 shutdown_connections 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6415cbb0 0x7fbc640cc4b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4031006421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6406f0a0 0x7fbc640cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- 172.21.15.28:0/4031006421 >> 172.21.15.28:0/4031006421 conn(0x7fbc640674f0 msgr2=0x7fbc6406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- 172.21.15.28:0/4031006421 shutdown_connections 2024-03-21T08:41:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- 172.21.15.28:0/4031006421 wait complete. 2024-03-21T08:41:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 Processor -- start 2024-03-21T08:41:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.786+0000 7fbc6c200640 1 -- start start 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 0x7fbc640b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6415cbb0 0x7fbc64138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64144b20 con 0x7fbc64071a80 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc640cce00 con 0x7fbc6406f0a0 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64168f20 con 0x7fbc6415cbb0 2024-03-21T08:41:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc69800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 0x7fbc640b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51270/0 (socket says 172.21.15.28:51270) 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 -- 172.21.15.28:0/4292265996 learned_addr learned my addr 172.21.15.28:0/4292265996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6a200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6415cbb0 0x7fbc64138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 -- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6415cbb0 msgr2=0x7fbc64138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6415cbb0 0x7fbc64138280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 -- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 msgr2=0x7fbc640b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 0x7fbc640b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 -- 172.21.15.28:0/4292265996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc640b2d00 con 0x7fbc64071a80 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc69800640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 0x7fbc640b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc68e00640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000e9b0 tx=0x7fbc5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50017070 con 0x7fbc64071a80 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5000c9a0 con 0x7fbc64071a80 2024-03-21T08:41:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc640b2fe0 con 0x7fbc64071a80 2024-03-21T08:41:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000cd70 con 0x7fbc64071a80 2024-03-21T08:41:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6413b4f0 con 0x7fbc64071a80 2024-03-21T08:41:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc64073430 con 0x7fbc64071a80 2024-03-21T08:41:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.790+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbc5001a070 con 0x7fbc64071a80 2024-03-21T08:41:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.794+0000 7fbc61600640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.794+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbc500865f0 con 0x7fbc64071a80 2024-03-21T08:41:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.794+0000 7fbc69800640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.794+0000 7fbc69800640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbc58007fd0 tx=0x7fbc58007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.798+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbc50056660 con 0x7fbc64071a80 2024-03-21T08:41:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.970+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbc6404f4f0 con 0x7fbc64071a80 2024-03-21T08:41:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.970+0000 7fbc61600640 1 -- 172.21.15.28:0/4292265996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbc5005a700 con 0x7fbc64071a80 2024-03-21T08:41:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 msgr2=0x7fbc4406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbc58007fd0 tx=0x7fbc58007d60 comp rx=0 tx=0).stop 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 msgr2=0x7fbc6413afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000e9b0 tx=0x7fbc5000ee70 comp rx=0 tx=0).stop 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 shutdown_connections 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6415cbb0 0x7fbc64138280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc64071a80 0x7fbc6413afb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 --2- 172.21.15.28:0/4292265996 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6406f0a0 0x7fbc640b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 >> 172.21.15.28:0/4292265996 conn(0x7fbc640674f0 msgr2=0x7fbc6413e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 shutdown_connections 2024-03-21T08:41:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:22.974+0000 7fbc6c200640 1 -- 172.21.15.28:0/4292265996 wait complete. 2024-03-21T08:41:22.997 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:41:22.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:22.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-21T08:41:23.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[32102]: fuse finished with error 0 and tester_r 0 2024-03-21T08:41:23.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:41:23.114 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:41:23.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:23.115 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-21T08:41:23.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.310+0000 7f15dc200640 1 Processor -- start 2024-03-21T08:41:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.310+0000 7f15dc200640 1 -- start start 2024-03-21T08:41:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.310+0000 7f15dc200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f15d40cb370 0x7f15d40cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.310+0000 7f15dc200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40cd9b0 con 0x7f15d4158760 2024-03-21T08:41:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15dc200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40cdc30 con 0x7f15d40d0510 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15dc200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d40ce1f0 con 0x7f15d40cb370 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f15d4158760 0x7f15d40cd2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36742/0 (socket says 172.21.15.28:36742) 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d9800640 1 -- 172.21.15.28:0/3948570764 learned_addr learned my addr 172.21.15.28:0/3948570764 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d8e00640 1 --2- 172.21.15.28:0/3948570764 >> v2:172.21.15.151:3300/0 conn(0x7f15d40cb370 0x7f15d40cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d3e00640 1 -- 172.21.15.28:0/3948570764 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699390488 0 0) 0x7f15c8001120 con 0x7f15d4158760 2024-03-21T08:41:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d3e00640 1 -- 172.21.15.28:0/3948570764 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc003530 con 0x7f15d4158760 2024-03-21T08:41:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d8e00640 1 -- 172.21.15.28:0/3948570764 >> v1:172.21.15.151:6789/0 conn(0x7f15d40d0510 legacy=0x7f15d40d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d8e00640 1 -- 172.21.15.28:0/3948570764 >> v1:172.21.15.28:6789/0 conn(0x7f15d4158760 legacy=0x7f15d40cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d8e00640 1 -- 172.21.15.28:0/3948570764 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40cdf10 con 0x7f15d40cb370 2024-03-21T08:41:23.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d8e00640 1 --2- 172.21.15.28:0/3948570764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d40cb800 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15c00097f0 tx=0x7f15c001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aebc11d4bf7fa559 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.314+0000 7f15d3e00640 1 -- 172.21.15.28:0/3948570764 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c00b1070 con 0x7f15d40cb370 2024-03-21T08:41:23.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d3e00640 1 -- 172.21.15.28:0/3948570764 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c00a8050 con 0x7f15d40cb370 2024-03-21T08:41:23.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d3e00640 1 -- 172.21.15.28:0/3948570764 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c00ac840 con 0x7f15d40cb370 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- 172.21.15.28:0/3948570764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 msgr2=0x7f15d40cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 --2- 172.21.15.28:0/3948570764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d40cb800 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15c00097f0 tx=0x7f15c001c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- 172.21.15.28:0/3948570764 shutdown_connections 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 --2- 172.21.15.28:0/3948570764 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d40cb800 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- 172.21.15.28:0/3948570764 >> 172.21.15.28:0/3948570764 conn(0x7f15d4099a30 msgr2=0x7f15d4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- 172.21.15.28:0/3948570764 shutdown_connections 2024-03-21T08:41:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- 172.21.15.28:0/3948570764 wait complete. 2024-03-21T08:41:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 Processor -- start 2024-03-21T08:41:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- start start 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15d4158760 0x7f15d40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d415b110 con 0x7f15d40d0510 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40cf7f0 con 0x7f15d40cb370 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15dc200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d415b290 con 0x7f15d4158760 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51276/0 (socket says 172.21.15.28:51276) 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d9800640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 -- 172.21.15.28:0/65390803 learned_addr learned my addr 172.21.15.28:0/65390803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15da200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15d4158760 0x7f15d40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 -- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15d4158760 msgr2=0x7f15d40c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15d4158760 0x7f15d40c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 -- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 msgr2=0x7f15d414b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d414b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.318+0000 7f15d8e00640 1 -- 172.21.15.28:0/65390803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d414bbd0 con 0x7f15d40d0510 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d8e00640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f15c00af040 tx=0x7f15c001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c00bd070 con 0x7f15d40d0510 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c00b2040 con 0x7f15d40d0510 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c00b1050 con 0x7f15d40d0510 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d40b5cb0 con 0x7f15d40d0510 2024-03-21T08:41:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d4157c60 con 0x7f15d40d0510 2024-03-21T08:41:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d404f410 con 0x7f15d40d0510 2024-03-21T08:41:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f15c00c0070 con 0x7f15d40d0510 2024-03-21T08:41:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 0x7f15b4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f15c011fdf0 con 0x7f15d40d0510 2024-03-21T08:41:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.322+0000 7f15d9800640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 0x7f15b4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.326+0000 7f15d9800640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 0x7f15b4069fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15c80012c0 tx=0x7f15c8007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.326+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f15d414bbd0 con 0x7f15d40d0510 2024-03-21T08:41:23.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.490+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15d40736a0 con 0x7f15d40d0510 2024-03-21T08:41:23.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.490+0000 7f15d1600640 1 -- 172.21.15.28:0/65390803 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f15c0016090 con 0x7f15d40d0510 2024-03-21T08:41:23.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:23.495 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:23.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 msgr2=0x7f15b4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 0x7f15b4069fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15c80012c0 tx=0x7f15c8007b80 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 msgr2=0x7f15d40b5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f15c00af040 tx=0x7f15c001ac20 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 shutdown_connections 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15b4067b20 0x7f15b4069fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15d4158760 0x7f15d40c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15d40d0510 0x7f15d40b5770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 --2- 172.21.15.28:0/65390803 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15d40cb370 0x7f15d414b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 >> 172.21.15.28:0/65390803 conn(0x7f15d4099a30 msgr2=0x7f15d40d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 shutdown_connections 2024-03-21T08:41:23.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.494+0000 7f15dc200640 1 -- 172.21.15.28:0/65390803 wait complete. 2024-03-21T08:41:23.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 Processor -- start 2024-03-21T08:41:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 -- start start 2024-03-21T08:41:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 0x7f32c806b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 0x7f32c806b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80cc1f0 con 0x7f32c815f540 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c815be00 con 0x7f32c80cf500 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cde00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80cfa00 con 0x7f32c815ba20 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32bfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 0x7f32c806b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cca00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 0x7f32c806b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cd400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f32c815f540 0x7f32c80ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39602/0 (socket says 172.21.15.28:39602) 2024-03-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cd400640 1 -- 172.21.15.28:0/644067681 learned_addr learned my addr 172.21.15.28:0/644067681 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32c7e00640 1 -- 172.21.15.28:0/644067681 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963714893 0 0) 0x7f32c00014a0 con 0x7f32c815f540 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32c7e00640 1 -- 172.21.15.28:0/644067681 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003530 con 0x7f32c815f540 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cca00640 1 -- 172.21.15.28:0/644067681 >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 msgr2=0x7f32c806b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cca00640 1 --2- 172.21.15.28:0/644067681 >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 0x7f32c806b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cca00640 1 -- 172.21.15.28:0/644067681 >> v1:172.21.15.151:6789/0 conn(0x7f32c815f540 legacy=0x7f32c80ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32cca00640 1 -- 172.21.15.28:0/644067681 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c80cfce0 con 0x7f32c815ba20 2024-03-21T08:41:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.654+0000 7f32bfe00640 1 --2- 172.21.15.28:0/644067681 >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 0x7f32c806b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cca00640 1 --2- 172.21.15.28:0/644067681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 0x7f32c806b170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097c0 tx=0x7f32b801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872aeaa2087b0f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32c7e00640 1 -- 172.21.15.28:0/644067681 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80a7070 con 0x7f32c815ba20 2024-03-21T08:41:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32c7e00640 1 -- 172.21.15.28:0/644067681 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b80b0070 con 0x7f32c815ba20 2024-03-21T08:41:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32c7e00640 1 -- 172.21.15.28:0/644067681 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80ab890 con 0x7f32c815ba20 2024-03-21T08:41:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 -- 172.21.15.28:0/644067681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 msgr2=0x7f32c806b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 --2- 172.21.15.28:0/644067681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 0x7f32c806b170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f32b80097c0 tx=0x7f32b801c470 comp rx=0 tx=0).stop 2024-03-21T08:41:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 -- 172.21.15.28:0/644067681 shutdown_connections 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 --2- 172.21.15.28:0/644067681 >> v2:172.21.15.28:3300/0 conn(0x7f32c80cf500 0x7f32c806b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 --2- 172.21.15.28:0/644067681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815ba20 0x7f32c806b170 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 -- 172.21.15.28:0/644067681 >> 172.21.15.28:0/644067681 conn(0x7f32c80b2280 msgr2=0x7f32c80b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 -- 172.21.15.28:0/644067681 shutdown_connections 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.658+0000 7f32cde00640 1 -- 172.21.15.28:0/644067681 wait complete. 2024-03-21T08:41:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 Processor -- start 2024-03-21T08:41:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- start start 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 0x7f32c815e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815f540 0x7f32c80b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c8168830 con 0x7f32c815ba20 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c81686b0 con 0x7f32c80cf500 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c81689b0 con 0x7f32c815f540 2024-03-21T08:41:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 0x7f32c815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:58924/0 (socket says 172.21.15.28:58924) 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 -- 172.21.15.28:0/3998201297 learned_addr learned my addr 172.21.15.28:0/3998201297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cd400640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815f540 0x7f32c80b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 -- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815f540 msgr2=0x7f32c80b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815f540 0x7f32c80b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 -- 172.21.15.28:0/3998201297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 msgr2=0x7f32c815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 0x7f32c815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 -- 172.21.15.28:0/3998201297 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c80b0d50 con 0x7f32c80cf500 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32bfe00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 0x7f32c815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cca00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f32b80ad040 tx=0x7f32b801acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80bf610 con 0x7f32c80cf500 2024-03-21T08:41:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b80bf7f0 con 0x7f32c80cf500 2024-03-21T08:41:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80bfbe0 con 0x7f32c80cf500 2024-03-21T08:41:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c815eca0 con 0x7f32c80cf500 2024-03-21T08:41:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.662+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c80c5140 con 0x7f32c80cf500 2024-03-21T08:41:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c8075270 con 0x7f32c80cf500 2024-03-21T08:41:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f32b80bd020 con 0x7f32c80cf500 2024-03-21T08:41:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32c5600640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 0x7f32a406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f32b8120ba0 con 0x7f32c80cf500 2024-03-21T08:41:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32bfe00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 0x7f32a406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.666+0000 7f32bfe00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 0x7f32a406a010 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009580 tx=0x7f32b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.670+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f32b8120e20 con 0x7f32c80cf500 2024-03-21T08:41:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.834+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32c80cde50 con 0x7f32c80cf500 2024-03-21T08:41:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.838+0000 7f32c5600640 1 -- 172.21.15.28:0/3998201297 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f32b8016090 con 0x7f32c80cf500 2024-03-21T08:41:23.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:23.842 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 msgr2=0x7f32a406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 0x7f32a406a010 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009580 tx=0x7f32b0009190 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 msgr2=0x7f32c815ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f32b80ad040 tx=0x7f32b801acb0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 shutdown_connections 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a4067b50 0x7f32a406a010 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c815f540 0x7f32c80b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ba20 0x7f32c815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 --2- 172.21.15.28:0/3998201297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32c80cf500 0x7f32c815ddf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 >> 172.21.15.28:0/3998201297 conn(0x7f32c80b2280 msgr2=0x7f32c80b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 shutdown_connections 2024-03-21T08:41:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:23.842+0000 7f32cde00640 1 -- 172.21.15.28:0/3998201297 wait complete. 2024-03-21T08:41:23.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-21T08:41:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 Processor -- start 2024-03-21T08:41:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 -- start start 2024-03-21T08:41:24.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f70700c50c0 0x7f70700ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f707015fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7070162940 con 0x7f70700c4ce0 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7070160320 con 0x7f70700c50c0 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f7076400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7070162c00 con 0x7f707015f970 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f70700c50c0 0x7f70700ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706f400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f70700c4ce0 0x7f707008b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39618/0 (socket says 172.21.15.28:39618) 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706f400640 1 -- 172.21.15.28:0/162583000 learned_addr learned my addr 172.21.15.28:0/162583000 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:24.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706fe00640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f707015fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706e000640 1 -- 172.21.15.28:0/162583000 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931432191 0 0) 0x7f7070162940 con 0x7f70700c4ce0 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706e000640 1 -- 172.21.15.28:0/162583000 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7054003530 con 0x7f70700c4ce0 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706ea00640 1 -- 172.21.15.28:0/162583000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 msgr2=0x7f707015fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706ea00640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f707015fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706ea00640 1 -- 172.21.15.28:0/162583000 >> v1:172.21.15.151:6789/0 conn(0x7f70700c4ce0 legacy=0x7f707008b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706ea00640 1 -- 172.21.15.28:0/162583000 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070162ee0 con 0x7f70700c50c0 2024-03-21T08:41:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.006+0000 7f706fe00640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f707015fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f706ea00640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f70700ce330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f705c0097c0 tx=0x7f705c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fad46f0136457a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f706e000640 1 -- 172.21.15.28:0/162583000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0b1070 con 0x7f70700c50c0 2024-03-21T08:41:24.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f706e000640 1 -- 172.21.15.28:0/162583000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c0a8050 con 0x7f70700c50c0 2024-03-21T08:41:24.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f706e000640 1 -- 172.21.15.28:0/162583000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0ac860 con 0x7f70700c50c0 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 -- 172.21.15.28:0/162583000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 msgr2=0x7f70700ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f70700ce330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f705c0097c0 tx=0x7f705c01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 -- 172.21.15.28:0/162583000 shutdown_connections 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f707015fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 --2- 172.21.15.28:0/162583000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f70700ce330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.010+0000 7f7076400640 1 -- 172.21.15.28:0/162583000 >> 172.21.15.28:0/162583000 conn(0x7f70700d1ab0 msgr2=0x7f70700d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- 172.21.15.28:0/162583000 shutdown_connections 2024-03-21T08:41:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- 172.21.15.28:0/162583000 wait complete. 2024-03-21T08:41:24.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 Processor -- start 2024-03-21T08:41:24.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- start start 2024-03-21T08:41:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 0x7f707015a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f7070164590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70700d1800 con 0x7f70700c50c0 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70700d1680 con 0x7f70700c4ce0 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f7076400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70700b2fd0 con 0x7f707015f970 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 0x7f707015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51316/0 (socket says 172.21.15.28:51316) 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 0x7f707015a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:58928/0 (socket says 172.21.15.28:58928) 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 -- 172.21.15.28:0/1261054894 learned_addr learned my addr 172.21.15.28:0/1261054894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f7070164590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 -- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 msgr2=0x7f7070164590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f7070164590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 -- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 msgr2=0x7f707015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 0x7f707015a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 -- 172.21.15.28:0/1261054894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700c9a50 con 0x7f70700c50c0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.014+0000 7f706ea00640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f705c0af040 tx=0x7f705c019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c0b1040 con 0x7f70700c50c0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c0bb070 con 0x7f70700c50c0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70700c9d30 con 0x7f70700c50c0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c006e20 con 0x7f70700c50c0 2024-03-21T08:41:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70700ca2f0 con 0x7f70700c50c0 2024-03-21T08:41:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70700b4610 con 0x7f70700c50c0 2024-03-21T08:41:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f705c0c0460 con 0x7f70700c50c0 2024-03-21T08:41:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 0x7f7050069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f705c11fd40 con 0x7f70700c50c0 2024-03-21T08:41:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.018+0000 7f706f400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 0x7f7050069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.022+0000 7f706f400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 0x7f7050069fe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70600011f0 tx=0x7f7060007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:24.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.022+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f705c0c5020 con 0x7f70700c50c0 2024-03-21T08:41:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:24.226+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f707004f4f0 con 0x7f70700c50c0 2024-03-21T08:41:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.066+0000 7f7067400640 1 -- 172.21.15.28:0/1261054894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f705c016090 con 0x7f70700c50c0 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 msgr2=0x7f7050069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 0x7f7050069fe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70600011f0 tx=0x7f7060007d40 comp rx=0 tx=0).stop 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 msgr2=0x7f7070161aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f705c0af040 tx=0x7f705c019180 comp rx=0 tx=0).stop 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 shutdown_connections 2024-03-21T08:41:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7050067b20 0x7f7050069fe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f707015f970 0x7f7070164590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70700c50c0 0x7f7070161aa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 --2- 172.21.15.28:0/1261054894 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70700c4ce0 0x7f707015a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 >> 172.21.15.28:0/1261054894 conn(0x7f70700d1ab0 msgr2=0x7f70700b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 shutdown_connections 2024-03-21T08:41:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.070+0000 7f7076400640 1 -- 172.21.15.28:0/1261054894 wait complete. 2024-03-21T08:41:25.098 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-21T08:41:25.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-21T08:41:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 Processor -- start 2024-03-21T08:41:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 -- start start 2024-03-21T08:41:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433806f0a0 0x7f43380cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433806f500 con 0x7f4338071a80 2024-03-21T08:41:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380ce5b0 con 0x7f433815cbb0 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.246+0000 7f433fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ce380 con 0x7f433806f0a0 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f433d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4338071a80 0x7f4338071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36746/0 (socket says 172.21.15.28:36746) 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f433d200640 1 -- 172.21.15.28:0/1980306707 learned_addr learned my addr 172.21.15.28:0/1980306707 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433806f0a0 0x7f43380cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807383697 0 0) 0x7f433806f500 con 0x7f4338071a80 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320003530 con 0x7f4338071a80 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040570502 0 0) 0x7f43380ce5b0 con 0x7f433815cbb0 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433806f500 con 0x7f433815cbb0 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496444635 0 0) 0x7f4320003530 con 0x7f4338071a80 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433806f0a0 msgr2=0x7f43380cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433806f0a0 0x7f43380cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 >> v1:172.21.15.151:6789/0 conn(0x7f433815cbb0 legacy=0x7f43380cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380ce890 con 0x7f4338071a80 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f432c001680 con 0x7f4338071a80 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433806f0a0 0x7f43380cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 >> v1:172.21.15.28:6789/0 conn(0x7f4338071a80 legacy=0x7f4338071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4320003b80 0x7f4320003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4320004810 0x7f4320004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43200051d0 0x7f432000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432000c010 con 0x7f4320004810 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432000c1c0 con 0x7f43200051d0 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432000c3c0 con 0x7f4320003b80 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4320004810 0x7f4320004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f433d200640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4320003b80 0x7f4320003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f433dc00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43200051d0 0x7f432000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 -- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4320003b80 msgr2=0x7f4320003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4320003b80 0x7f4320003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 -- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43200051d0 msgr2=0x7f432000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f43200051d0 0x7f432000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f4337e00640 1 -- 172.21.15.28:0/1980306707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320003530 con 0x7f4320004810 2024-03-21T08:41:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.250+0000 7f433d200640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4320003b80 0x7f4320003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f4337e00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4320004810 0x7f4320004c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4338072f20 tx=0x7f432401caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a491a4c61981269 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324009810 con 0x7f4320004810 2024-03-21T08:41:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324004030 con 0x7f4320004810 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f4337400640 1 -- 172.21.15.28:0/1980306707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240043e0 con 0x7f4320004810 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- 172.21.15.28:0/1980306707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4320004810 msgr2=0x7f4320004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 --2- 172.21.15.28:0/1980306707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4320004810 0x7f4320004c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4338072f20 tx=0x7f432401caf0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433dc00640 1 -- 172.21.15.28:0/1980306707 reap_dead start 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- 172.21.15.28:0/1980306707 shutdown_connections 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- 172.21.15.28:0/1980306707 >> 172.21.15.28:0/1980306707 conn(0x7f43380674f0 msgr2=0x7f433806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- 172.21.15.28:0/1980306707 shutdown_connections 2024-03-21T08:41:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- 172.21.15.28:0/1980306707 wait complete. 2024-03-21T08:41:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 Processor -- start 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- start start 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4320004810 0x7f43381426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 0x7f43381619d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380cdfc0 con 0x7f433813cf10 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380cde40 con 0x7f4320004810 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.254+0000 7f433fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ce140 con 0x7f433813d860 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51332/0 (socket says 172.21.15.28:51332) 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 -- 172.21.15.28:0/2407123660 learned_addr learned my addr 172.21.15.28:0/2407123660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433d200640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4320004810 0x7f43381426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433dc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 0x7f43381619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 -- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 msgr2=0x7f43381619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 0x7f43381619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 -- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4320004810 msgr2=0x7f43381426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4320004810 0x7f43381426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 -- 172.21.15.28:0/2407123660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433813dd00 con 0x7f433813cf10 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433dc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 0x7f43381619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4337e00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43240982f0 tx=0x7f4320005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240bf070 con 0x7f433813cf10 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324003620 con 0x7f433813cf10 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324003a10 con 0x7f433813cf10 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433813dfe0 con 0x7f433813cf10 2024-03-21T08:41:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4338142190 con 0x7f433813cf10 2024-03-21T08:41:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4338073430 con 0x7f433813cf10 2024-03-21T08:41:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f43240c2070 con 0x7f433813cf10 2024-03-21T08:41:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 0x7f4318069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.258+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4324120240 con 0x7f433813cf10 2024-03-21T08:41:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.262+0000 7f433d200640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 0x7f4318069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.262+0000 7f433d200640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 0x7f4318069fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c001e60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.262+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4324016800 con 0x7f433813cf10 2024-03-21T08:41:25.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:25.462+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f43380b0c50 con 0x7f433813cf10 2024-03-21T08:41:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.070+0000 7f4334c00640 1 -- 172.21.15.28:0/2407123660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f43240f0220 con 0x7f433813cf10 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 msgr2=0x7f4318069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 0x7f4318069fe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c001e60 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 msgr2=0x7f433813d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f43240982f0 tx=0x7f4320005840 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 shutdown_connections 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4318067b20 0x7f4318069fe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f433813d860 0x7f43381619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f433813cf10 0x7f433813d320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 --2- 172.21.15.28:0/2407123660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4320004810 0x7f43381426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 >> 172.21.15.28:0/2407123660 conn(0x7f43380674f0 msgr2=0x7f433806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 shutdown_connections 2024-03-21T08:41:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.074+0000 7f433fc00640 1 -- 172.21.15.28:0/2407123660 wait complete. 2024-03-21T08:41:26.102 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-21T08:41:26.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.242+0000 7f0a72e00640 1 Processor -- start 2024-03-21T08:41:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.242+0000 7f0a72e00640 1 -- start start 2024-03-21T08:41:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a72e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a6c0cd380 0x7f0a6c0b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a72e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c15bf00 con 0x7f0a6c0b5520 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a72e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c0cdad0 con 0x7f0a6c0af060 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a72e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b3a30 con 0x7f0a6c0cd380 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a63e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a6c0b5520 0x7f0a6c15c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36756/0 (socket says 172.21.15.28:36756) 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a63e00640 1 -- 172.21.15.28:0/1588130126 learned_addr learned my addr 172.21.15.28:0/1588130126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a70e00640 1 --2- 172.21.15.28:0/1588130126 >> v2:172.21.15.28:3300/0 conn(0x7f0a6c0cd380 0x7f0a6c0b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a70e00640 1 -- 172.21.15.28:0/1588130126 >> v1:172.21.15.151:6789/0 conn(0x7f0a6c0af060 legacy=0x7f0a6c0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a70e00640 1 -- 172.21.15.28:0/1588130126 >> v1:172.21.15.28:6789/0 conn(0x7f0a6c0b5520 legacy=0x7f0a6c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a70e00640 1 -- 172.21.15.28:0/1588130126 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0cddb0 con 0x7f0a6c0cd380 2024-03-21T08:41:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.246+0000 7f0a6b400640 1 -- 172.21.15.28:0/1588130126 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068793433 0 0) 0x7f0a6c15bf00 con 0x7f0a6c0b5520 2024-03-21T08:41:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a70e00640 1 --2- 172.21.15.28:0/1588130126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c0b34f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009d90 tx=0x7f0a6401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365ee8a4487c281f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a6b400640 1 -- 172.21.15.28:0/1588130126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640b2070 con 0x7f0a6c0cd380 2024-03-21T08:41:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a6b400640 1 -- 172.21.15.28:0/1588130126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a640a5890 con 0x7f0a6c0cd380 2024-03-21T08:41:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a6b400640 1 -- 172.21.15.28:0/1588130126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640a5c60 con 0x7f0a6c0cd380 2024-03-21T08:41:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- 172.21.15.28:0/1588130126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0cd380 msgr2=0x7f0a6c0b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 --2- 172.21.15.28:0/1588130126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c0b34f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009d90 tx=0x7f0a6401c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- 172.21.15.28:0/1588130126 shutdown_connections 2024-03-21T08:41:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 --2- 172.21.15.28:0/1588130126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c0b34f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- 172.21.15.28:0/1588130126 >> 172.21.15.28:0/1588130126 conn(0x7f0a6c0cc230 msgr2=0x7f0a6c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- 172.21.15.28:0/1588130126 shutdown_connections 2024-03-21T08:41:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- 172.21.15.28:0/1588130126 wait complete. 2024-03-21T08:41:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 Processor -- start 2024-03-21T08:41:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.250+0000 7f0a72e00640 1 -- start start 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a6c14a2d0 0x7f0a6c0ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c15d2a0 con 0x7f0a6c0b5520 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c15d120 con 0x7f0a6c0cd380 2024-03-21T08:41:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c15d420 con 0x7f0a6c14a2d0 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51344/0 (socket says 172.21.15.28:51344) 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 -- 172.21.15.28:0/2598001852 learned_addr learned my addr 172.21.15.28:0/2598001852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a63e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a70e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a6c14a2d0 0x7f0a6c0ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 -- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a6c14a2d0 msgr2=0x7f0a6c0ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a6c14a2d0 0x7f0a6c0ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 -- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a6c0cd380 msgr2=0x7f0a6c149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 -- 172.21.15.28:0/2598001852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c06e300 con 0x7f0a6c0b5520 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a6be00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c500 tx=0x7f0a5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c005220 con 0x7f0a6c0b5520 2024-03-21T08:41:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0053c0 con 0x7f0a6c0b5520 2024-03-21T08:41:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c007270 con 0x7f0a6c0b5520 2024-03-21T08:41:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c06f7e0 con 0x7f0a6c0b5520 2024-03-21T08:41:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c06e850 con 0x7f0a6c0b5520 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c04f410 con 0x7f0a6c0b5520 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.254+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0a5c019020 con 0x7f0a6c0b5520 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.258+0000 7f0a68c00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 0x7f0a4806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.258+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0a5c085400 con 0x7f0a6c0b5520 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.258+0000 7f0a63e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 0x7f0a4806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.258+0000 7f0a63e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 0x7f0a4806a010 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c1fa1b0 tx=0x7f0a54007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.262+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0a5c0553f0 con 0x7f0a6c0b5520 2024-03-21T08:41:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a6c0ccd50 con 0x7f0a6c0b5520 2024-03-21T08:41:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a68c00640 1 -- 172.21.15.28:0/2598001852 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f0a5c059490 con 0x7f0a6c0b5520 2024-03-21T08:41:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:26.461 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 msgr2=0x7f0a4806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 0x7f0a4806a010 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c1fa1b0 tx=0x7f0a54007240 comp rx=0 tx=0).stop 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 msgr2=0x7f0a6c146e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c500 tx=0x7f0a5c00c9c0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 shutdown_connections 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0a48067b50 0x7f0a4806a010 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0a6c14a2d0 0x7f0a6c0ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0a6c0cd380 0x7f0a6c149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 --2- 172.21.15.28:0/2598001852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a6c0b5520 0x7f0a6c146e70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 >> 172.21.15.28:0/2598001852 conn(0x7f0a6c0cc230 msgr2=0x7f0a6c073b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 shutdown_connections 2024-03-21T08:41:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.458+0000 7f0a72e00640 1 -- 172.21.15.28:0/2598001852 wait complete. 2024-03-21T08:41:26.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 Processor -- start 2024-03-21T08:41:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 -- start start 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdca40aff70 0x7fdca406b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 0x7fdca406b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40d0540 con 0x7fdca415dd00 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40cd6e0 con 0x7fdca40aff70 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcac600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40cd9c0 con 0x7fdca406eb30 2024-03-21T08:41:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdca40aff70 0x7fdca406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdca415dd00 0x7fdca40cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36760/0 (socket says 172.21.15.28:36760) 2024-03-21T08:41:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9c00640 1 -- 172.21.15.28:0/3549126760 learned_addr learned my addr 172.21.15.28:0/3549126760 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcaa600640 1 --2- 172.21.15.28:0/3549126760 >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 0x7fdca406b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9200640 1 -- 172.21.15.28:0/3549126760 >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 msgr2=0x7fdca406b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9200640 1 --2- 172.21.15.28:0/3549126760 >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 0x7fdca406b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9200640 1 -- 172.21.15.28:0/3549126760 >> v1:172.21.15.28:6789/0 conn(0x7fdca415dd00 legacy=0x7fdca40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdca9200640 1 -- 172.21.15.28:0/3549126760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40cdca0 con 0x7fdca40aff70 2024-03-21T08:41:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.622+0000 7fdcaa600640 1 --2- 172.21.15.28:0/3549126760 >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 0x7fdca406b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdca9200640 1 --2- 172.21.15.28:0/3549126760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca40aff70 0x7fdca406b170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdca40cf0a0 tx=0x7fdc9001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16877c1cd7ad1025 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdca3e00640 1 -- 172.21.15.28:0/3549126760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc900b1070 con 0x7fdca40aff70 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdca3e00640 1 -- 172.21.15.28:0/3549126760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc900a8050 con 0x7fdca40aff70 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdca3e00640 1 -- 172.21.15.28:0/3549126760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc900ac860 con 0x7fdca40aff70 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 -- 172.21.15.28:0/3549126760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca40aff70 msgr2=0x7fdca406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 --2- 172.21.15.28:0/3549126760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca40aff70 0x7fdca406b170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdca40cf0a0 tx=0x7fdc9001c450 comp rx=0 tx=0).stop 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 -- 172.21.15.28:0/3549126760 shutdown_connections 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 --2- 172.21.15.28:0/3549126760 >> v2:172.21.15.151:3300/0 conn(0x7fdca406eb30 0x7fdca406b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 --2- 172.21.15.28:0/3549126760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca40aff70 0x7fdca406b170 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 -- 172.21.15.28:0/3549126760 >> 172.21.15.28:0/3549126760 conn(0x7fdca4099a30 msgr2=0x7fdca4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 -- 172.21.15.28:0/3549126760 shutdown_connections 2024-03-21T08:41:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.626+0000 7fdcac600640 1 -- 172.21.15.28:0/3549126760 wait complete. 2024-03-21T08:41:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 Processor -- start 2024-03-21T08:41:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- start start 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdca40aff70 0x7fdca41467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 0x7fdca4146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4162240 con 0x7fdca406eb30 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca415e2f0 con 0x7fdca40aff70 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca41623c0 con 0x7fdca415dd00 2024-03-21T08:41:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51358/0 (socket says 172.21.15.28:51358) 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 -- 172.21.15.28:0/4152268089 learned_addr learned my addr 172.21.15.28:0/4152268089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9200640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdca40aff70 0x7fdca41467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcaa600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 0x7fdca4146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 -- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 msgr2=0x7fdca4146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 0x7fdca4146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 -- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdca40aff70 msgr2=0x7fdca41467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdca40aff70 0x7fdca41467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 -- 172.21.15.28:0/4152268089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca415e6f0 con 0x7fdca406eb30 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcaa600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 0x7fdca4146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca9c00640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c370 tx=0x7fdc9800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98017070 con 0x7fdca406eb30 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc98005cf0 con 0x7fdca406eb30 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca415e9d0 con 0x7fdca406eb30 2024-03-21T08:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc980072f0 con 0x7fdca406eb30 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca415ef90 con 0x7fdca406eb30 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.630+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca404f410 con 0x7fdca406eb30 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.634+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fdc9801a070 con 0x7fdca406eb30 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.634+0000 7fdca1600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 0x7fdc84069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.634+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdc980854f0 con 0x7fdca406eb30 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.634+0000 7fdca9200640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 0x7fdc84069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.634+0000 7fdca9200640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 0x7fdc84069fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc900ae000 tx=0x7fdc90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.638+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdc980555e0 con 0x7fdca406eb30 2024-03-21T08:41:26.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.802+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdca4157100 con 0x7fdca406eb30 2024-03-21T08:41:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.802+0000 7fdca1600640 1 -- 172.21.15.28:0/4152268089 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7fdc98059680 con 0x7fdca406eb30 2024-03-21T08:41:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:26.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 msgr2=0x7fdc84069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 0x7fdc84069fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc900ae000 tx=0x7fdc90008040 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 msgr2=0x7fdca415bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c370 tx=0x7fdc9800c830 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 shutdown_connections 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdc84067b20 0x7fdc84069fe0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdca415dd00 0x7fdca4146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdca40aff70 0x7fdca41467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 --2- 172.21.15.28:0/4152268089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca406eb30 0x7fdca415bfb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 >> 172.21.15.28:0/4152268089 conn(0x7fdca4099a30 msgr2=0x7fdca40af560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 shutdown_connections 2024-03-21T08:41:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.806+0000 7fdcac600640 1 -- 172.21.15.28:0/4152268089 wait complete. 2024-03-21T08:41:26.827 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-21T08:41:26.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:41:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 Processor -- start 2024-03-21T08:41:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 -- start start 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7b2415f260 0x7f7b240cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240cdb50 con 0x7f7b2409a290 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240cde30 con 0x7f7b240cd5d0 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b2aa00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b240c5530 con 0x7f7b2415f260 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b28a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7b2409a290 0x7f7b240c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36772/0 (socket says 172.21.15.28:36772) 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b28a00640 1 -- 172.21.15.28:0/822690278 learned_addr learned my addr 172.21.15.28:0/822690278 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23e00640 1 --2- 172.21.15.28:0/822690278 >> v2:172.21.15.151:3300/0 conn(0x7f7b2415f260 0x7f7b240cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251776575 0 0) 0x7f7b240cdb50 con 0x7f7b2409a290 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08003530 con 0x7f7b2409a290 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680099021 0 0) 0x7f7b240cde30 con 0x7f7b240cd5d0 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23e00640 1 -- 172.21.15.28:0/822690278 >> v1:172.21.15.151:6789/0 conn(0x7f7b240cd5d0 legacy=0x7f7b24099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23e00640 1 -- 172.21.15.28:0/822690278 >> v1:172.21.15.28:6789/0 conn(0x7f7b2409a290 legacy=0x7f7b240c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.966+0000 7f7b23e00640 1 -- 172.21.15.28:0/822690278 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240c5780 con 0x7f7b2415f260 2024-03-21T08:41:26.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b23e00640 1 --2- 172.21.15.28:0/822690278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2415f260 0x7f7b240cd090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7b140097c0 tx=0x7f7b1401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a18ce819cd94889 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b140ab600 con 0x7f7b2415f260 2024-03-21T08:41:26.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b140b0070 con 0x7f7b2415f260 2024-03-21T08:41:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b23400640 1 -- 172.21.15.28:0/822690278 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b140ab9c0 con 0x7f7b2415f260 2024-03-21T08:41:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 -- 172.21.15.28:0/822690278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2415f260 msgr2=0x7f7b240cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/822690278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2415f260 0x7f7b240cd090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7b140097c0 tx=0x7f7b1401c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 -- 172.21.15.28:0/822690278 shutdown_connections 2024-03-21T08:41:26.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/822690278 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2415f260 0x7f7b240cd090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 -- 172.21.15.28:0/822690278 >> 172.21.15.28:0/822690278 conn(0x7f7b240b34d0 msgr2=0x7f7b240b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:26.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 -- 172.21.15.28:0/822690278 shutdown_connections 2024-03-21T08:41:26.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.970+0000 7f7b2aa00640 1 -- 172.21.15.28:0/822690278 wait complete. 2024-03-21T08:41:26.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 Processor -- start 2024-03-21T08:41:26.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- start start 2024-03-21T08:41:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2409a290 0x7f7b2415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b240cd5d0 0x7f7b24163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c7020 con 0x7f7b2415f260 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c6ea0 con 0x7f7b2409a290 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240c71a0 con 0x7f7b240cd5d0 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51370/0 (socket says 172.21.15.28:51370) 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 -- 172.21.15.28:0/40497758 learned_addr learned my addr 172.21.15.28:0/40497758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b1be00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b240cd5d0 0x7f7b24163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b23e00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2409a290 0x7f7b2415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 -- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b240cd5d0 msgr2=0x7f7b24163c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b240cd5d0 0x7f7b24163c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 -- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2409a290 msgr2=0x7f7b2415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2409a290 0x7f7b2415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 -- 172.21.15.28:0/40497758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2415cb90 con 0x7f7b2415f260 2024-03-21T08:41:26.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b28a00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00cae0 tx=0x7f7b1c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c017070 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00ac40 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b241675e0 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c012640 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b241649e0 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.974+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2404f410 con 0x7f7b2415f260 2024-03-21T08:41:26.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.978+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7b1c01a070 con 0x7f7b2415f260 2024-03-21T08:41:26.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.978+0000 7f7b20c00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 0x7f7b0006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:26.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.978+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7b1c085f20 con 0x7f7b2415f260 2024-03-21T08:41:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.978+0000 7f7b23e00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 0x7f7b0006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:26.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.978+0000 7f7b23e00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 0x7f7b0006a010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b140af000 tx=0x7f7b140a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:26.982+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7b2415cb90 con 0x7f7b2415f260 2024-03-21T08:41:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.142+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b24099e30 con 0x7f7b00067b50 2024-03-21T08:41:27.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b20c00640 1 -- 172.21.15.28:0/40497758 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332562 (secure 0 0 0) 0x7f7b24099e30 con 0x7f7b00067b50 2024-03-21T08:41:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:27.163 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:41:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 msgr2=0x7f7b0006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 0x7f7b0006a010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b140af000 tx=0x7f7b140a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 msgr2=0x7f7b24070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00cae0 tx=0x7f7b1c00a060 comp rx=0 tx=0).stop 2024-03-21T08:41:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 shutdown_connections 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b2415f260 0x7f7b24070e70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7b00067b50 0x7f7b0006a010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7b240cd5d0 0x7f7b24163c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 --2- 172.21.15.28:0/40497758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b2409a290 0x7f7b2415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 >> 172.21.15.28:0/40497758 conn(0x7f7b240b34d0 msgr2=0x7f7b240701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 shutdown_connections 2024-03-21T08:41:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.158+0000 7f7b2aa00640 1 -- 172.21.15.28:0/40497758 wait complete. 2024-03-21T08:41:27.181 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-21T08:41:25.722503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231892,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742828,"statfs":{"total":773094113280,"available":772856655872,"internally_reserved":0,"allocated":3317760,"data_stored":2007752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161451+0000","last_change":"2024-03-21T08:41:07.021438+0000","last_active":"2024-03-21T08:41:13.161451+0000","last_peered":"2024-03-21T08:41:13.161451+0000","last_clean":"2024-03-21T08:41:13.161451+0000","last_became_active":"2024-03-21T08:41:07.021142+0000","last_became_peered":"2024-03-21T08:41:07.021142+0000","last_unstale":"2024-03-21T08:41:13.161451+0000","last_undegraded":"2024-03-21T08:41:13.161451+0000","last_fullsized":"2024-03-21T08:41:13.161451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:15:32.936778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262630+0000","last_change":"2024-03-21T08:41:08.010424+0000","last_active":"2024-03-21T08:41:14.262630+0000","last_peered":"2024-03-21T08:41:14.262630+0000","last_clean":"2024-03-21T08:41:14.262630+0000","last_became_active":"2024-03-21T08:41:08.010169+0000","last_became_peered":"2024-03-21T08:41:08.010169+0000","last_unstale":"2024-03-21T08:41:14.262630+0000","last_undegraded":"2024-03-21T08:41:14.262630+0000","last_fullsized":"2024-03-21T08:41:14.262630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261510+0000","last_change":"2024-03-21T08:41:12.340891+0000","last_active":"2024-03-21T08:41:14.261510+0000","last_peered":"2024-03-21T08:41:14.261510+0000","last_clean":"2024-03-21T08:41:14.261510+0000","last_became_active":"2024-03-21T08:41:12.340181+0000","last_became_peered":"2024-03-21T08:41:12.340181+0000","last_unstale":"2024-03-21T08:41:14.261510+0000","last_undegraded":"2024-03-21T08:41:14.261510+0000","last_fullsized":"2024-03-21T08:41:14.261510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262757+0000","last_change":"2024-03-21T08:41:07.009640+0000","last_active":"2024-03-21T08:41:14.262757+0000","last_peered":"2024-03-21T08:41:14.262757+0000","last_clean":"2024-03-21T08:41:14.262757+0000","last_became_active":"2024-03-21T08:41:07.009390+0000","last_became_peered":"2024-03-21T08:41:07.009390+0000","last_unstale":"2024-03-21T08:41:14.262757+0000","last_undegraded":"2024-03-21T08:41:14.262757+0000","last_fullsized":"2024-03-21T08:41:14.262757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161250+0000","last_change":"2024-03-21T08:41:08.010008+0000","last_active":"2024-03-21T08:41:13.161250+0000","last_peered":"2024-03-21T08:41:13.161250+0000","last_clean":"2024-03-21T08:41:13.161250+0000","last_became_active":"2024-03-21T08:41:08.008163+0000","last_became_peered":"2024-03-21T08:41:08.008163+0000","last_unstale":"2024-03-21T08:41:13.161250+0000","last_undegraded":"2024-03-21T08:41:13.161250+0000","last_fullsized":"2024-03-21T08:41:13.161250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025449+0000","last_change":"2024-03-21T08:41:12.049570+0000","last_active":"2024-03-21T08:41:13.025449+0000","last_peered":"2024-03-21T08:41:13.025449+0000","last_clean":"2024-03-21T08:41:13.025449+0000","last_became_active":"2024-03-21T08:41:12.048903+0000","last_became_peered":"2024-03-21T08:41:12.048903+0000","last_unstale":"2024-03-21T08:41:13.025449+0000","last_undegraded":"2024-03-21T08:41:13.025449+0000","last_fullsized":"2024-03-21T08:41:13.025449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:10.935038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736993+0000","last_change":"2024-03-21T08:41:07.022988+0000","last_active":"2024-03-21T08:41:13.736993+0000","last_peered":"2024-03-21T08:41:13.736993+0000","last_clean":"2024-03-21T08:41:13.736993+0000","last_became_active":"2024-03-21T08:41:07.022643+0000","last_became_peered":"2024-03-21T08:41:07.022643+0000","last_unstale":"2024-03-21T08:41:13.736993+0000","last_undegraded":"2024-03-21T08:41:13.736993+0000","last_fullsized":"2024-03-21T08:41:13.736993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260935+0000","last_change":"2024-03-21T08:41:08.008250+0000","last_active":"2024-03-21T08:41:14.260935+0000","last_peered":"2024-03-21T08:41:14.260935+0000","last_clean":"2024-03-21T08:41:14.260935+0000","last_became_active":"2024-03-21T08:41:08.007638+0000","last_became_peered":"2024-03-21T08:41:08.007638+0000","last_unstale":"2024-03-21T08:41:14.260935+0000","last_undegraded":"2024-03-21T08:41:14.260935+0000","last_fullsized":"2024-03-21T08:41:14.260935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:14.189872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027230+0000","last_change":"2024-03-21T08:41:12.051136+0000","last_active":"2024-03-21T08:41:13.027230+0000","last_peered":"2024-03-21T08:41:13.027230+0000","last_clean":"2024-03-21T08:41:13.027230+0000","last_became_active":"2024-03-21T08:41:12.050706+0000","last_became_peered":"2024-03-21T08:41:12.050706+0000","last_unstale":"2024-03-21T08:41:13.027230+0000","last_undegraded":"2024-03-21T08:41:13.027230+0000","last_fullsized":"2024-03-21T08:41:13.027230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:28:03.320473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736865+0000","last_change":"2024-03-21T08:41:07.021316+0000","last_active":"2024-03-21T08:41:13.736865+0000","last_peered":"2024-03-21T08:41:13.736865+0000","last_clean":"2024-03-21T08:41:13.736865+0000","last_became_active":"2024-03-21T08:41:07.021039+0000","last_became_peered":"2024-03-21T08:41:07.021039+0000","last_unstale":"2024-03-21T08:41:13.736865+0000","last_undegraded":"2024-03-21T08:41:13.736865+0000","last_fullsized":"2024-03-21T08:41:13.736865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261060+0000","last_change":"2024-03-21T08:41:08.004273+0000","last_active":"2024-03-21T08:41:14.261060+0000","last_peered":"2024-03-21T08:41:14.261060+0000","last_clean":"2024-03-21T08:41:14.261060+0000","last_became_active":"2024-03-21T08:41:08.003800+0000","last_became_peered":"2024-03-21T08:41:08.003800+0000","last_unstale":"2024-03-21T08:41:14.261060+0000","last_undegraded":"2024-03-21T08:41:14.261060+0000","last_fullsized":"2024-03-21T08:41:14.261060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027771+0000","last_change":"2024-03-21T08:41:12.046634+0000","last_active":"2024-03-21T08:41:13.027771+0000","last_peered":"2024-03-21T08:41:13.027771+0000","last_clean":"2024-03-21T08:41:13.027771+0000","last_became_active":"2024-03-21T08:41:12.046224+0000","last_became_peered":"2024-03-21T08:41:12.046224+0000","last_unstale":"2024-03-21T08:41:13.027771+0000","last_undegraded":"2024-03-21T08:41:13.027771+0000","last_fullsized":"2024-03-21T08:41:13.027771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:20.757184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025520+0000","last_change":"2024-03-21T08:41:08.009427+0000","last_active":"2024-03-21T08:41:13.025520+0000","last_peered":"2024-03-21T08:41:13.025520+0000","last_clean":"2024-03-21T08:41:13.025520+0000","last_became_active":"2024-03-21T08:41:08.009082+0000","last_became_peered":"2024-03-21T08:41:08.009082+0000","last_unstale":"2024-03-21T08:41:13.025520+0000","last_undegraded":"2024-03-21T08:41:13.025520+0000","last_fullsized":"2024-03-21T08:41:13.025520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:07.174703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736774+0000","last_change":"2024-03-21T08:41:07.019968+0000","last_active":"2024-03-21T08:41:13.736774+0000","last_peered":"2024-03-21T08:41:13.736774+0000","last_clean":"2024-03-21T08:41:13.736774+0000","last_became_active":"2024-03-21T08:41:07.019039+0000","last_became_peered":"2024-03-21T08:41:07.019039+0000","last_unstale":"2024-03-21T08:41:13.736774+0000","last_undegraded":"2024-03-21T08:41:13.736774+0000","last_fullsized":"2024-03-21T08:41:13.736774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027696+0000","last_change":"2024-03-21T08:41:12.053357+0000","last_active":"2024-03-21T08:41:13.027696+0000","last_peered":"2024-03-21T08:41:13.027696+0000","last_clean":"2024-03-21T08:41:13.027696+0000","last_became_active":"2024-03-21T08:41:12.053045+0000","last_became_peered":"2024-03-21T08:41:12.053045+0000","last_unstale":"2024-03-21T08:41:13.027696+0000","last_undegraded":"2024-03-21T08:41:13.027696+0000","last_fullsized":"2024-03-21T08:41:13.027696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:52.060002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262174+0000","last_change":"2024-03-21T08:41:08.014032+0000","last_active":"2024-03-21T08:41:14.262174+0000","last_peered":"2024-03-21T08:41:14.262174+0000","last_clean":"2024-03-21T08:41:14.262174+0000","last_became_active":"2024-03-21T08:41:08.013716+0000","last_became_peered":"2024-03-21T08:41:08.013716+0000","last_unstale":"2024-03-21T08:41:14.262174+0000","last_undegraded":"2024-03-21T08:41:14.262174+0000","last_fullsized":"2024-03-21T08:41:14.262174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262296+0000","last_change":"2024-03-21T08:41:07.008347+0000","last_active":"2024-03-21T08:41:14.262296+0000","last_peered":"2024-03-21T08:41:14.262296+0000","last_clean":"2024-03-21T08:41:14.262296+0000","last_became_active":"2024-03-21T08:41:07.005441+0000","last_became_peered":"2024-03-21T08:41:07.005441+0000","last_unstale":"2024-03-21T08:41:14.262296+0000","last_undegraded":"2024-03-21T08:41:14.262296+0000","last_fullsized":"2024-03-21T08:41:14.262296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025374+0000","last_change":"2024-03-21T08:41:12.051147+0000","last_active":"2024-03-21T08:41:13.025374+0000","last_peered":"2024-03-21T08:41:13.025374+0000","last_clean":"2024-03-21T08:41:13.025374+0000","last_became_active":"2024-03-21T08:41:12.050744+0000","last_became_peered":"2024-03-21T08:41:12.050744+0000","last_unstale":"2024-03-21T08:41:13.025374+0000","last_undegraded":"2024-03-21T08:41:13.025374+0000","last_fullsized":"2024-03-21T08:41:13.025374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:15.657762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261625+0000","last_change":"2024-03-21T08:41:08.003261+0000","last_active":"2024-03-21T08:41:14.261625+0000","last_peered":"2024-03-21T08:41:14.261625+0000","last_clean":"2024-03-21T08:41:14.261625+0000","last_became_active":"2024-03-21T08:41:08.002934+0000","last_became_peered":"2024-03-21T08:41:08.002934+0000","last_unstale":"2024-03-21T08:41:14.261625+0000","last_undegraded":"2024-03-21T08:41:14.261625+0000","last_fullsized":"2024-03-21T08:41:14.261625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262016+0000","last_change":"2024-03-21T08:41:07.020788+0000","last_active":"2024-03-21T08:41:14.262016+0000","last_peered":"2024-03-21T08:41:14.262016+0000","last_clean":"2024-03-21T08:41:14.262016+0000","last_became_active":"2024-03-21T08:41:07.020341+0000","last_became_peered":"2024-03-21T08:41:07.020341+0000","last_unstale":"2024-03-21T08:41:14.262016+0000","last_undegraded":"2024-03-21T08:41:14.262016+0000","last_fullsized":"2024-03-21T08:41:14.262016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263514+0000","last_change":"2024-03-21T08:41:12.340482+0000","last_active":"2024-03-21T08:41:14.263514+0000","last_peered":"2024-03-21T08:41:14.263514+0000","last_clean":"2024-03-21T08:41:14.263514+0000","last_became_active":"2024-03-21T08:41:12.340076+0000","last_became_peered":"2024-03-21T08:41:12.340076+0000","last_unstale":"2024-03-21T08:41:14.263514+0000","last_undegraded":"2024-03-21T08:41:14.263514+0000","last_fullsized":"2024-03-21T08:41:14.263514+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:10.484672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027247+0000","last_change":"2024-03-21T08:41:08.002403+0000","last_active":"2024-03-21T08:41:13.027247+0000","last_peered":"2024-03-21T08:41:13.027247+0000","last_clean":"2024-03-21T08:41:13.027247+0000","last_became_active":"2024-03-21T08:41:08.001933+0000","last_became_peered":"2024-03-21T08:41:08.001933+0000","last_unstale":"2024-03-21T08:41:13.027247+0000","last_undegraded":"2024-03-21T08:41:13.027247+0000","last_fullsized":"2024-03-21T08:41:13.027247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:40.121158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736405+0000","last_change":"2024-03-21T08:41:07.019255+0000","last_active":"2024-03-21T08:41:13.736405+0000","last_peered":"2024-03-21T08:41:13.736405+0000","last_clean":"2024-03-21T08:41:13.736405+0000","last_became_active":"2024-03-21T08:41:07.018958+0000","last_became_peered":"2024-03-21T08:41:07.018958+0000","last_unstale":"2024-03-21T08:41:13.736405+0000","last_undegraded":"2024-03-21T08:41:13.736405+0000","last_fullsized":"2024-03-21T08:41:13.736405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027872+0000","last_change":"2024-03-21T08:41:12.049104+0000","last_active":"2024-03-21T08:41:13.027872+0000","last_peered":"2024-03-21T08:41:13.027872+0000","last_clean":"2024-03-21T08:41:13.027872+0000","last_became_active":"2024-03-21T08:41:12.048724+0000","last_became_peered":"2024-03-21T08:41:12.048724+0000","last_unstale":"2024-03-21T08:41:13.027872+0000","last_undegraded":"2024-03-21T08:41:13.027872+0000","last_fullsized":"2024-03-21T08:41:13.027872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161500+0000","last_change":"2024-03-21T08:41:08.006976+0000","last_active":"2024-03-21T08:41:13.161500+0000","last_peered":"2024-03-21T08:41:13.161500+0000","last_clean":"2024-03-21T08:41:13.161500+0000","last_became_active":"2024-03-21T08:41:08.006565+0000","last_became_peered":"2024-03-21T08:41:08.006565+0000","last_unstale":"2024-03-21T08:41:13.161500+0000","last_undegraded":"2024-03-21T08:41:13.161500+0000","last_fullsized":"2024-03-21T08:41:13.161500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026772+0000","last_change":"2024-03-21T08:41:07.014131+0000","last_active":"2024-03-21T08:41:13.026772+0000","last_peered":"2024-03-21T08:41:13.026772+0000","last_clean":"2024-03-21T08:41:13.026772+0000","last_became_active":"2024-03-21T08:41:07.013789+0000","last_became_peered":"2024-03-21T08:41:07.013789+0000","last_unstale":"2024-03-21T08:41:13.026772+0000","last_undegraded":"2024-03-21T08:41:13.026772+0000","last_fullsized":"2024-03-21T08:41:13.026772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027502+0000","last_change":"2024-03-21T08:41:12.336087+0000","last_active":"2024-03-21T08:41:13.027502+0000","last_peered":"2024-03-21T08:41:13.027502+0000","last_clean":"2024-03-21T08:41:13.027502+0000","last_became_active":"2024-03-21T08:41:12.335722+0000","last_became_peered":"2024-03-21T08:41:12.335722+0000","last_unstale":"2024-03-21T08:41:13.027502+0000","last_undegraded":"2024-03-21T08:41:13.027502+0000","last_fullsized":"2024-03-21T08:41:13.027502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:47.106105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262994+0000","last_change":"2024-03-21T08:41:08.021861+0000","last_active":"2024-03-21T08:41:14.262994+0000","last_peered":"2024-03-21T08:41:14.262994+0000","last_clean":"2024-03-21T08:41:14.262994+0000","last_became_active":"2024-03-21T08:41:08.021564+0000","last_became_peered":"2024-03-21T08:41:08.021564+0000","last_unstale":"2024-03-21T08:41:14.262994+0000","last_undegraded":"2024-03-21T08:41:14.262994+0000","last_fullsized":"2024-03-21T08:41:14.262994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263418+0000","last_change":"2024-03-21T08:41:07.010549+0000","last_active":"2024-03-21T08:41:14.263418+0000","last_peered":"2024-03-21T08:41:14.263418+0000","last_clean":"2024-03-21T08:41:14.263418+0000","last_became_active":"2024-03-21T08:41:07.010129+0000","last_became_peered":"2024-03-21T08:41:07.010129+0000","last_unstale":"2024-03-21T08:41:14.263418+0000","last_undegraded":"2024-03-21T08:41:14.263418+0000","last_fullsized":"2024-03-21T08:41:14.263418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261368+0000","last_change":"2024-03-21T08:41:12.340111+0000","last_active":"2024-03-21T08:41:14.261368+0000","last_peered":"2024-03-21T08:41:14.261368+0000","last_clean":"2024-03-21T08:41:14.261368+0000","last_became_active":"2024-03-21T08:41:12.339714+0000","last_became_peered":"2024-03-21T08:41:12.339714+0000","last_unstale":"2024-03-21T08:41:14.261368+0000","last_undegraded":"2024-03-21T08:41:14.261368+0000","last_fullsized":"2024-03-21T08:41:14.261368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:30:09.350945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262982+0000","last_change":"2024-03-21T08:41:08.005617+0000","last_active":"2024-03-21T08:41:14.262982+0000","last_peered":"2024-03-21T08:41:14.262982+0000","last_clean":"2024-03-21T08:41:14.262982+0000","last_became_active":"2024-03-21T08:41:08.005254+0000","last_became_peered":"2024-03-21T08:41:08.005254+0000","last_unstale":"2024-03-21T08:41:14.262982+0000","last_undegraded":"2024-03-21T08:41:14.262982+0000","last_fullsized":"2024-03-21T08:41:14.262982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737093+0000","last_change":"2024-03-21T08:41:07.019857+0000","last_active":"2024-03-21T08:41:13.737093+0000","last_peered":"2024-03-21T08:41:13.737093+0000","last_clean":"2024-03-21T08:41:13.737093+0000","last_became_active":"2024-03-21T08:41:07.019553+0000","last_became_peered":"2024-03-21T08:41:07.019553+0000","last_unstale":"2024-03-21T08:41:13.737093+0000","last_undegraded":"2024-03-21T08:41:13.737093+0000","last_fullsized":"2024-03-21T08:41:13.737093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:35:34.563261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161134+0000","last_change":"2024-03-21T08:41:12.335655+0000","last_active":"2024-03-21T08:41:13.161134+0000","last_peered":"2024-03-21T08:41:13.161134+0000","last_clean":"2024-03-21T08:41:13.161134+0000","last_became_active":"2024-03-21T08:41:12.335219+0000","last_became_peered":"2024-03-21T08:41:12.335219+0000","last_unstale":"2024-03-21T08:41:13.161134+0000","last_undegraded":"2024-03-21T08:41:13.161134+0000","last_fullsized":"2024-03-21T08:41:13.161134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261158+0000","last_change":"2024-03-21T08:41:08.016121+0000","last_active":"2024-03-21T08:41:14.261158+0000","last_peered":"2024-03-21T08:41:14.261158+0000","last_clean":"2024-03-21T08:41:14.261158+0000","last_became_active":"2024-03-21T08:41:08.015237+0000","last_became_peered":"2024-03-21T08:41:08.015237+0000","last_unstale":"2024-03-21T08:41:14.261158+0000","last_undegraded":"2024-03-21T08:41:14.261158+0000","last_fullsized":"2024-03-21T08:41:14.261158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:48.028184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263069+0000","last_change":"2024-03-21T08:41:07.013927+0000","last_active":"2024-03-21T08:41:14.263069+0000","last_peered":"2024-03-21T08:41:14.263069+0000","last_clean":"2024-03-21T08:41:14.263069+0000","last_became_active":"2024-03-21T08:41:07.013595+0000","last_became_peered":"2024-03-21T08:41:07.013595+0000","last_unstale":"2024-03-21T08:41:14.263069+0000","last_undegraded":"2024-03-21T08:41:14.263069+0000","last_fullsized":"2024-03-21T08:41:14.263069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262181+0000","last_change":"2024-03-21T08:41:12.056397+0000","last_active":"2024-03-21T08:41:14.262181+0000","last_peered":"2024-03-21T08:41:14.262181+0000","last_clean":"2024-03-21T08:41:14.262181+0000","last_became_active":"2024-03-21T08:41:12.054349+0000","last_became_peered":"2024-03-21T08:41:12.054349+0000","last_unstale":"2024-03-21T08:41:14.262181+0000","last_undegraded":"2024-03-21T08:41:14.262181+0000","last_fullsized":"2024-03-21T08:41:14.262181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027232+0000","last_change":"2024-03-21T08:41:07.997069+0000","last_active":"2024-03-21T08:41:13.027232+0000","last_peered":"2024-03-21T08:41:13.027232+0000","last_clean":"2024-03-21T08:41:13.027232+0000","last_became_active":"2024-03-21T08:41:07.996691+0000","last_became_peered":"2024-03-21T08:41:07.996691+0000","last_unstale":"2024-03-21T08:41:13.027232+0000","last_undegraded":"2024-03-21T08:41:13.027232+0000","last_fullsized":"2024-03-21T08:41:13.027232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:59.860172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260862+0000","last_change":"2024-03-21T08:41:07.004740+0000","last_active":"2024-03-21T08:41:14.260862+0000","last_peered":"2024-03-21T08:41:14.260862+0000","last_clean":"2024-03-21T08:41:14.260862+0000","last_became_active":"2024-03-21T08:41:07.004315+0000","last_became_peered":"2024-03-21T08:41:07.004315+0000","last_unstale":"2024-03-21T08:41:14.260862+0000","last_undegraded":"2024-03-21T08:41:14.260862+0000","last_fullsized":"2024-03-21T08:41:14.260862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:58.854457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263063+0000","last_change":"2024-03-21T08:41:12.049583+0000","last_active":"2024-03-21T08:41:14.263063+0000","last_peered":"2024-03-21T08:41:14.263063+0000","last_clean":"2024-03-21T08:41:14.263063+0000","last_became_active":"2024-03-21T08:41:12.048879+0000","last_became_peered":"2024-03-21T08:41:12.048879+0000","last_unstale":"2024-03-21T08:41:14.263063+0000","last_undegraded":"2024-03-21T08:41:14.263063+0000","last_fullsized":"2024-03-21T08:41:14.263063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:41.204023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261754+0000","last_change":"2024-03-21T08:41:08.006302+0000","last_active":"2024-03-21T08:41:14.261754+0000","last_peered":"2024-03-21T08:41:14.261754+0000","last_clean":"2024-03-21T08:41:14.261754+0000","last_became_active":"2024-03-21T08:41:08.005255+0000","last_became_peered":"2024-03-21T08:41:08.005255+0000","last_unstale":"2024-03-21T08:41:14.261754+0000","last_undegraded":"2024-03-21T08:41:14.261754+0000","last_fullsized":"2024-03-21T08:41:14.261754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025568+0000","last_change":"2024-03-21T08:41:07.004853+0000","last_active":"2024-03-21T08:41:13.025568+0000","last_peered":"2024-03-21T08:41:13.025568+0000","last_clean":"2024-03-21T08:41:13.025568+0000","last_became_active":"2024-03-21T08:41:07.004447+0000","last_became_peered":"2024-03-21T08:41:07.004447+0000","last_unstale":"2024-03-21T08:41:13.025568+0000","last_undegraded":"2024-03-21T08:41:13.025568+0000","last_fullsized":"2024-03-21T08:41:13.025568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263561+0000","last_change":"2024-03-21T08:41:12.057098+0000","last_active":"2024-03-21T08:41:14.263561+0000","last_peered":"2024-03-21T08:41:14.263561+0000","last_clean":"2024-03-21T08:41:14.263561+0000","last_became_active":"2024-03-21T08:41:12.056306+0000","last_became_peered":"2024-03-21T08:41:12.056306+0000","last_unstale":"2024-03-21T08:41:14.263561+0000","last_undegraded":"2024-03-21T08:41:14.263561+0000","last_fullsized":"2024-03-21T08:41:14.263561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:00:00.666748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160776+0000","last_change":"2024-03-21T08:41:07.995187+0000","last_active":"2024-03-21T08:41:13.160776+0000","last_peered":"2024-03-21T08:41:13.160776+0000","last_clean":"2024-03-21T08:41:13.160776+0000","last_became_active":"2024-03-21T08:41:07.994882+0000","last_became_peered":"2024-03-21T08:41:07.994882+0000","last_unstale":"2024-03-21T08:41:13.160776+0000","last_undegraded":"2024-03-21T08:41:13.160776+0000","last_fullsized":"2024-03-21T08:41:13.160776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:27.177399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736421+0000","last_change":"2024-03-21T08:41:07.019906+0000","last_active":"2024-03-21T08:41:13.736421+0000","last_peered":"2024-03-21T08:41:13.736421+0000","last_clean":"2024-03-21T08:41:13.736421+0000","last_became_active":"2024-03-21T08:41:07.019585+0000","last_became_peered":"2024-03-21T08:41:07.019585+0000","last_unstale":"2024-03-21T08:41:13.736421+0000","last_undegraded":"2024-03-21T08:41:13.736421+0000","last_fullsized":"2024-03-21T08:41:13.736421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161357+0000","last_change":"2024-03-21T08:41:12.337710+0000","last_active":"2024-03-21T08:41:13.161357+0000","last_peered":"2024-03-21T08:41:13.161357+0000","last_clean":"2024-03-21T08:41:13.161357+0000","last_became_active":"2024-03-21T08:41:12.336270+0000","last_became_peered":"2024-03-21T08:41:12.336270+0000","last_unstale":"2024-03-21T08:41:13.161357+0000","last_undegraded":"2024-03-21T08:41:13.161357+0000","last_fullsized":"2024-03-21T08:41:13.161357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:11.100710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261759+0000","last_change":"2024-03-21T08:41:08.015395+0000","last_active":"2024-03-21T08:41:14.261759+0000","last_peered":"2024-03-21T08:41:14.261759+0000","last_clean":"2024-03-21T08:41:14.261759+0000","last_became_active":"2024-03-21T08:41:08.015091+0000","last_became_peered":"2024-03-21T08:41:08.015091+0000","last_unstale":"2024-03-21T08:41:14.261759+0000","last_undegraded":"2024-03-21T08:41:14.261759+0000","last_fullsized":"2024-03-21T08:41:14.261759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260786+0000","last_change":"2024-03-21T08:41:07.008427+0000","last_active":"2024-03-21T08:41:14.260786+0000","last_peered":"2024-03-21T08:41:14.260786+0000","last_clean":"2024-03-21T08:41:14.260786+0000","last_became_active":"2024-03-21T08:41:07.008115+0000","last_became_peered":"2024-03-21T08:41:07.008115+0000","last_unstale":"2024-03-21T08:41:14.260786+0000","last_undegraded":"2024-03-21T08:41:14.260786+0000","last_fullsized":"2024-03-21T08:41:14.260786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736516+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:41:13.736516+0000","last_peered":"2024-03-21T08:41:13.736516+0000","last_clean":"2024-03-21T08:41:13.736516+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:41:13.736516+0000","last_undegraded":"2024-03-21T08:41:13.736516+0000","last_fullsized":"2024-03-21T08:41:13.736516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026534+0000","last_change":"2024-03-21T08:41:12.334882+0000","last_active":"2024-03-21T08:41:13.026534+0000","last_peered":"2024-03-21T08:41:13.026534+0000","last_clean":"2024-03-21T08:41:13.026534+0000","last_became_active":"2024-03-21T08:41:12.334459+0000","last_became_peered":"2024-03-21T08:41:12.334459+0000","last_unstale":"2024-03-21T08:41:13.026534+0000","last_undegraded":"2024-03-21T08:41:13.026534+0000","last_fullsized":"2024-03-21T08:41:13.026534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:17.097499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736898+0000","last_change":"2024-03-21T08:41:08.030635+0000","last_active":"2024-03-21T08:41:13.736898+0000","last_peered":"2024-03-21T08:41:13.736898+0000","last_clean":"2024-03-21T08:41:13.736898+0000","last_became_active":"2024-03-21T08:41:08.028635+0000","last_became_peered":"2024-03-21T08:41:08.028635+0000","last_unstale":"2024-03-21T08:41:13.736898+0000","last_undegraded":"2024-03-21T08:41:13.736898+0000","last_fullsized":"2024-03-21T08:41:13.736898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:14.013088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026041+0000","last_change":"2024-03-21T08:41:07.004839+0000","last_active":"2024-03-21T08:41:13.026041+0000","last_peered":"2024-03-21T08:41:13.026041+0000","last_clean":"2024-03-21T08:41:13.026041+0000","last_became_active":"2024-03-21T08:41:07.004409+0000","last_became_peered":"2024-03-21T08:41:07.004409+0000","last_unstale":"2024-03-21T08:41:13.026041+0000","last_undegraded":"2024-03-21T08:41:13.026041+0000","last_fullsized":"2024-03-21T08:41:13.026041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025597+0000","last_change":"2024-03-21T08:41:12.334998+0000","last_active":"2024-03-21T08:41:13.025597+0000","last_peered":"2024-03-21T08:41:13.025597+0000","last_clean":"2024-03-21T08:41:13.025597+0000","last_became_active":"2024-03-21T08:41:12.334610+0000","last_became_peered":"2024-03-21T08:41:12.334610+0000","last_unstale":"2024-03-21T08:41:13.025597+0000","last_undegraded":"2024-03-21T08:41:13.025597+0000","last_fullsized":"2024-03-21T08:41:13.025597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:04.540298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261002+0000","last_change":"2024-03-21T08:41:08.014545+0000","last_active":"2024-03-21T08:41:14.261002+0000","last_peered":"2024-03-21T08:41:14.261002+0000","last_clean":"2024-03-21T08:41:14.261002+0000","last_became_active":"2024-03-21T08:41:08.013815+0000","last_became_peered":"2024-03-21T08:41:08.013815+0000","last_unstale":"2024-03-21T08:41:14.261002+0000","last_undegraded":"2024-03-21T08:41:14.261002+0000","last_fullsized":"2024-03-21T08:41:14.261002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:07.068892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263256+0000","last_change":"2024-03-21T08:41:07.011021+0000","last_active":"2024-03-21T08:41:14.263256+0000","last_peered":"2024-03-21T08:41:14.263256+0000","last_clean":"2024-03-21T08:41:14.263256+0000","last_became_active":"2024-03-21T08:41:07.010411+0000","last_became_peered":"2024-03-21T08:41:07.010411+0000","last_unstale":"2024-03-21T08:41:14.263256+0000","last_undegraded":"2024-03-21T08:41:14.263256+0000","last_fullsized":"2024-03-21T08:41:14.263256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161403+0000","last_change":"2024-03-21T08:41:12.055570+0000","last_active":"2024-03-21T08:41:13.161403+0000","last_peered":"2024-03-21T08:41:13.161403+0000","last_clean":"2024-03-21T08:41:13.161403+0000","last_became_active":"2024-03-21T08:41:12.055174+0000","last_became_peered":"2024-03-21T08:41:12.055174+0000","last_unstale":"2024-03-21T08:41:13.161403+0000","last_undegraded":"2024-03-21T08:41:13.161403+0000","last_fullsized":"2024-03-21T08:41:13.161403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:45:15.642543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737253+0000","last_change":"2024-03-21T08:41:08.037018+0000","last_active":"2024-03-21T08:41:13.737253+0000","last_peered":"2024-03-21T08:41:13.737253+0000","last_clean":"2024-03-21T08:41:13.737253+0000","last_became_active":"2024-03-21T08:41:08.036676+0000","last_became_peered":"2024-03-21T08:41:08.036676+0000","last_unstale":"2024-03-21T08:41:13.737253+0000","last_undegraded":"2024-03-21T08:41:13.737253+0000","last_fullsized":"2024-03-21T08:41:13.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:59:56.634930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025975+0000","last_change":"2024-03-21T08:41:07.004711+0000","last_active":"2024-03-21T08:41:13.025975+0000","last_peered":"2024-03-21T08:41:13.025975+0000","last_clean":"2024-03-21T08:41:13.025975+0000","last_became_active":"2024-03-21T08:41:07.004192+0000","last_became_peered":"2024-03-21T08:41:07.004192+0000","last_unstale":"2024-03-21T08:41:13.025975+0000","last_undegraded":"2024-03-21T08:41:13.025975+0000","last_fullsized":"2024-03-21T08:41:13.025975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260477+0000","last_change":"2024-03-21T08:41:12.341167+0000","last_active":"2024-03-21T08:41:14.260477+0000","last_peered":"2024-03-21T08:41:14.260477+0000","last_clean":"2024-03-21T08:41:14.260477+0000","last_became_active":"2024-03-21T08:41:12.339602+0000","last_became_peered":"2024-03-21T08:41:12.339602+0000","last_unstale":"2024-03-21T08:41:14.260477+0000","last_undegraded":"2024-03-21T08:41:14.260477+0000","last_fullsized":"2024-03-21T08:41:14.260477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737088+0000","last_change":"2024-03-21T08:41:08.033751+0000","last_active":"2024-03-21T08:41:13.737088+0000","last_peered":"2024-03-21T08:41:13.737088+0000","last_clean":"2024-03-21T08:41:13.737088+0000","last_became_active":"2024-03-21T08:41:08.033458+0000","last_became_peered":"2024-03-21T08:41:08.033458+0000","last_unstale":"2024-03-21T08:41:13.737088+0000","last_undegraded":"2024-03-21T08:41:13.737088+0000","last_fullsized":"2024-03-21T08:41:13.737088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:13.065681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027669+0000","last_change":"2024-03-21T08:41:07.017025+0000","last_active":"2024-03-21T08:41:13.027669+0000","last_peered":"2024-03-21T08:41:13.027669+0000","last_clean":"2024-03-21T08:41:13.027669+0000","last_became_active":"2024-03-21T08:41:07.016654+0000","last_became_peered":"2024-03-21T08:41:07.016654+0000","last_unstale":"2024-03-21T08:41:13.027669+0000","last_undegraded":"2024-03-21T08:41:13.027669+0000","last_fullsized":"2024-03-21T08:41:13.027669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:10:03.436446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025227+0000","last_change":"2024-03-21T08:41:12.048497+0000","last_active":"2024-03-21T08:41:13.025227+0000","last_peered":"2024-03-21T08:41:13.025227+0000","last_clean":"2024-03-21T08:41:13.025227+0000","last_became_active":"2024-03-21T08:41:12.047987+0000","last_became_peered":"2024-03-21T08:41:12.047987+0000","last_unstale":"2024-03-21T08:41:13.025227+0000","last_undegraded":"2024-03-21T08:41:13.025227+0000","last_fullsized":"2024-03-21T08:41:13.025227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:39.659897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736537+0000","last_change":"2024-03-21T08:41:08.033999+0000","last_active":"2024-03-21T08:41:13.736537+0000","last_peered":"2024-03-21T08:41:13.736537+0000","last_clean":"2024-03-21T08:41:13.736537+0000","last_became_active":"2024-03-21T08:41:08.033015+0000","last_became_peered":"2024-03-21T08:41:08.033015+0000","last_unstale":"2024-03-21T08:41:13.736537+0000","last_undegraded":"2024-03-21T08:41:13.736537+0000","last_fullsized":"2024-03-21T08:41:13.736537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:16:25.591202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025409+0000","last_change":"2024-03-21T08:41:07.001332+0000","last_active":"2024-03-21T08:41:13.025409+0000","last_peered":"2024-03-21T08:41:13.025409+0000","last_clean":"2024-03-21T08:41:13.025409+0000","last_became_active":"2024-03-21T08:41:06.999605+0000","last_became_peered":"2024-03-21T08:41:06.999605+0000","last_unstale":"2024-03-21T08:41:13.025409+0000","last_undegraded":"2024-03-21T08:41:13.025409+0000","last_fullsized":"2024-03-21T08:41:13.025409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:06:31.096538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161359+0000","last_change":"2024-03-21T08:41:12.059105+0000","last_active":"2024-03-21T08:41:13.161359+0000","last_peered":"2024-03-21T08:41:13.161359+0000","last_clean":"2024-03-21T08:41:13.161359+0000","last_became_active":"2024-03-21T08:41:12.058643+0000","last_became_peered":"2024-03-21T08:41:12.058643+0000","last_unstale":"2024-03-21T08:41:13.161359+0000","last_undegraded":"2024-03-21T08:41:13.161359+0000","last_fullsized":"2024-03-21T08:41:13.161359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.221689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025534+0000","last_change":"2024-03-21T08:41:08.010609+0000","last_active":"2024-03-21T08:41:13.025534+0000","last_peered":"2024-03-21T08:41:13.025534+0000","last_clean":"2024-03-21T08:41:13.025534+0000","last_became_active":"2024-03-21T08:41:08.010109+0000","last_became_peered":"2024-03-21T08:41:08.010109+0000","last_unstale":"2024-03-21T08:41:13.025534+0000","last_undegraded":"2024-03-21T08:41:13.025534+0000","last_fullsized":"2024-03-21T08:41:13.025534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026017+0000","last_change":"2024-03-21T08:41:07.010384+0000","last_active":"2024-03-21T08:41:13.026017+0000","last_peered":"2024-03-21T08:41:13.026017+0000","last_clean":"2024-03-21T08:41:13.026017+0000","last_became_active":"2024-03-21T08:41:07.010019+0000","last_became_peered":"2024-03-21T08:41:07.010019+0000","last_unstale":"2024-03-21T08:41:13.026017+0000","last_undegraded":"2024-03-21T08:41:13.026017+0000","last_fullsized":"2024-03-21T08:41:13.026017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262930+0000","last_change":"2024-03-21T08:41:12.341603+0000","last_active":"2024-03-21T08:41:14.262930+0000","last_peered":"2024-03-21T08:41:14.262930+0000","last_clean":"2024-03-21T08:41:14.262930+0000","last_became_active":"2024-03-21T08:41:12.341158+0000","last_became_peered":"2024-03-21T08:41:12.341158+0000","last_unstale":"2024-03-21T08:41:14.262930+0000","last_undegraded":"2024-03-21T08:41:14.262930+0000","last_fullsized":"2024-03-21T08:41:14.262930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:02.246810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736756+0000","last_change":"2024-03-21T08:41:08.032505+0000","last_active":"2024-03-21T08:41:13.736756+0000","last_peered":"2024-03-21T08:41:13.736756+0000","last_clean":"2024-03-21T08:41:13.736756+0000","last_became_active":"2024-03-21T08:41:08.032026+0000","last_became_peered":"2024-03-21T08:41:08.032026+0000","last_unstale":"2024-03-21T08:41:13.736756+0000","last_undegraded":"2024-03-21T08:41:13.736756+0000","last_fullsized":"2024-03-21T08:41:13.736756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:21.306665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260844+0000","last_change":"2024-03-21T08:41:07.003264+0000","last_active":"2024-03-21T08:41:14.260844+0000","last_peered":"2024-03-21T08:41:14.260844+0000","last_clean":"2024-03-21T08:41:14.260844+0000","last_became_active":"2024-03-21T08:41:07.002969+0000","last_became_peered":"2024-03-21T08:41:07.002969+0000","last_unstale":"2024-03-21T08:41:14.260844+0000","last_undegraded":"2024-03-21T08:41:14.260844+0000","last_fullsized":"2024-03-21T08:41:14.260844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:19.390331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262423+0000","last_change":"2024-03-21T08:41:12.336949+0000","last_active":"2024-03-21T08:41:14.262423+0000","last_peered":"2024-03-21T08:41:14.262423+0000","last_clean":"2024-03-21T08:41:14.262423+0000","last_became_active":"2024-03-21T08:41:12.335654+0000","last_became_peered":"2024-03-21T08:41:12.335654+0000","last_unstale":"2024-03-21T08:41:14.262423+0000","last_undegraded":"2024-03-21T08:41:14.262423+0000","last_fullsized":"2024-03-21T08:41:14.262423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:54:40.677358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160969+0000","last_change":"2024-03-21T08:41:08.020852+0000","last_active":"2024-03-21T08:41:13.160969+0000","last_peered":"2024-03-21T08:41:13.160969+0000","last_clean":"2024-03-21T08:41:13.160969+0000","last_became_active":"2024-03-21T08:41:08.020547+0000","last_became_peered":"2024-03-21T08:41:08.020547+0000","last_unstale":"2024-03-21T08:41:13.160969+0000","last_undegraded":"2024-03-21T08:41:13.160969+0000","last_fullsized":"2024-03-21T08:41:13.160969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:56.518303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161459+0000","last_change":"2024-03-21T08:41:07.013979+0000","last_active":"2024-03-21T08:41:13.161459+0000","last_peered":"2024-03-21T08:41:13.161459+0000","last_clean":"2024-03-21T08:41:13.161459+0000","last_became_active":"2024-03-21T08:41:07.013725+0000","last_became_peered":"2024-03-21T08:41:07.013725+0000","last_unstale":"2024-03-21T08:41:13.161459+0000","last_undegraded":"2024-03-21T08:41:13.161459+0000","last_fullsized":"2024-03-21T08:41:13.161459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:03:13.978443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261111+0000","last_change":"2024-03-21T08:41:12.335008+0000","last_active":"2024-03-21T08:41:14.261111+0000","last_peered":"2024-03-21T08:41:14.261111+0000","last_clean":"2024-03-21T08:41:14.261111+0000","last_became_active":"2024-03-21T08:41:12.334635+0000","last_became_peered":"2024-03-21T08:41:12.334635+0000","last_unstale":"2024-03-21T08:41:14.261111+0000","last_undegraded":"2024-03-21T08:41:14.261111+0000","last_fullsized":"2024-03-21T08:41:14.261111+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736888+0000","last_change":"2024-03-21T08:41:08.026214+0000","last_active":"2024-03-21T08:41:13.736888+0000","last_peered":"2024-03-21T08:41:13.736888+0000","last_clean":"2024-03-21T08:41:13.736888+0000","last_became_active":"2024-03-21T08:41:08.025686+0000","last_became_peered":"2024-03-21T08:41:08.025686+0000","last_unstale":"2024-03-21T08:41:13.736888+0000","last_undegraded":"2024-03-21T08:41:13.736888+0000","last_fullsized":"2024-03-21T08:41:13.736888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:32:26.184390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:23.028503+0000","last_change":"2024-03-21T08:41:07.005099+0000","last_active":"2024-03-21T08:41:23.028503+0000","last_peered":"2024-03-21T08:41:23.028503+0000","last_clean":"2024-03-21T08:41:23.028503+0000","last_became_active":"2024-03-21T08:41:07.004825+0000","last_became_peered":"2024-03-21T08:41:07.004825+0000","last_unstale":"2024-03-21T08:41:23.028503+0000","last_undegraded":"2024-03-21T08:41:23.028503+0000","last_fullsized":"2024-03-21T08:41:23.028503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262224+0000","last_change":"2024-03-21T08:41:12.057213+0000","last_active":"2024-03-21T08:41:14.262224+0000","last_peered":"2024-03-21T08:41:14.262224+0000","last_clean":"2024-03-21T08:41:14.262224+0000","last_became_active":"2024-03-21T08:41:12.056539+0000","last_became_peered":"2024-03-21T08:41:12.056539+0000","last_unstale":"2024-03-21T08:41:14.262224+0000","last_undegraded":"2024-03-21T08:41:14.262224+0000","last_fullsized":"2024-03-21T08:41:14.262224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:35.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027129+0000","last_change":"2024-03-21T08:41:08.018576+0000","last_active":"2024-03-21T08:41:13.027129+0000","last_peered":"2024-03-21T08:41:13.027129+0000","last_clean":"2024-03-21T08:41:13.027129+0000","last_became_active":"2024-03-21T08:41:08.018277+0000","last_became_peered":"2024-03-21T08:41:08.018277+0000","last_unstale":"2024-03-21T08:41:13.027129+0000","last_undegraded":"2024-03-21T08:41:13.027129+0000","last_fullsized":"2024-03-21T08:41:13.027129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:00:26.730038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263165+0000","last_change":"2024-03-21T08:41:07.013500+0000","last_active":"2024-03-21T08:41:14.263165+0000","last_peered":"2024-03-21T08:41:14.263165+0000","last_clean":"2024-03-21T08:41:14.263165+0000","last_became_active":"2024-03-21T08:41:07.010794+0000","last_became_peered":"2024-03-21T08:41:07.010794+0000","last_unstale":"2024-03-21T08:41:14.263165+0000","last_undegraded":"2024-03-21T08:41:14.263165+0000","last_fullsized":"2024-03-21T08:41:14.263165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027024+0000","last_change":"2024-03-21T08:41:12.049156+0000","last_active":"2024-03-21T08:41:13.027024+0000","last_peered":"2024-03-21T08:41:13.027024+0000","last_clean":"2024-03-21T08:41:13.027024+0000","last_became_active":"2024-03-21T08:41:12.048737+0000","last_became_peered":"2024-03-21T08:41:12.048737+0000","last_unstale":"2024-03-21T08:41:13.027024+0000","last_undegraded":"2024-03-21T08:41:13.027024+0000","last_fullsized":"2024-03-21T08:41:13.027024+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:12.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161355+0000","last_change":"2024-03-21T08:41:08.012883+0000","last_active":"2024-03-21T08:41:13.161355+0000","last_peered":"2024-03-21T08:41:13.161355+0000","last_clean":"2024-03-21T08:41:13.161355+0000","last_became_active":"2024-03-21T08:41:08.012565+0000","last_became_peered":"2024-03-21T08:41:08.012565+0000","last_unstale":"2024-03-21T08:41:13.161355+0000","last_undegraded":"2024-03-21T08:41:13.161355+0000","last_fullsized":"2024-03-21T08:41:13.161355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261096+0000","last_change":"2024-03-21T08:41:07.013115+0000","last_active":"2024-03-21T08:41:14.261096+0000","last_peered":"2024-03-21T08:41:14.261096+0000","last_clean":"2024-03-21T08:41:14.261096+0000","last_became_active":"2024-03-21T08:41:07.012884+0000","last_became_peered":"2024-03-21T08:41:07.012884+0000","last_unstale":"2024-03-21T08:41:14.261096+0000","last_undegraded":"2024-03-21T08:41:14.261096+0000","last_fullsized":"2024-03-21T08:41:14.261096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:39.115443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261540+0000","last_change":"2024-03-21T08:41:12.051266+0000","last_active":"2024-03-21T08:41:14.261540+0000","last_peered":"2024-03-21T08:41:14.261540+0000","last_clean":"2024-03-21T08:41:14.261540+0000","last_became_active":"2024-03-21T08:41:12.050452+0000","last_became_peered":"2024-03-21T08:41:12.050452+0000","last_unstale":"2024-03-21T08:41:14.261540+0000","last_undegraded":"2024-03-21T08:41:14.261540+0000","last_fullsized":"2024-03-21T08:41:14.261540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261053+0000","last_change":"2024-03-21T08:41:08.029184+0000","last_active":"2024-03-21T08:41:14.261053+0000","last_peered":"2024-03-21T08:41:14.261053+0000","last_clean":"2024-03-21T08:41:14.261053+0000","last_became_active":"2024-03-21T08:41:08.028897+0000","last_became_peered":"2024-03-21T08:41:08.028897+0000","last_unstale":"2024-03-21T08:41:14.261053+0000","last_undegraded":"2024-03-21T08:41:14.261053+0000","last_fullsized":"2024-03-21T08:41:14.261053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:59.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026681+0000","last_change":"2024-03-21T08:41:07.021159+0000","last_active":"2024-03-21T08:41:13.026681+0000","last_peered":"2024-03-21T08:41:13.026681+0000","last_clean":"2024-03-21T08:41:13.026681+0000","last_became_active":"2024-03-21T08:41:07.020879+0000","last_became_peered":"2024-03-21T08:41:07.020879+0000","last_unstale":"2024-03-21T08:41:13.026681+0000","last_undegraded":"2024-03-21T08:41:13.026681+0000","last_fullsized":"2024-03-21T08:41:13.026681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261676+0000","last_change":"2024-03-21T08:41:12.340577+0000","last_active":"2024-03-21T08:41:14.261676+0000","last_peered":"2024-03-21T08:41:14.261676+0000","last_clean":"2024-03-21T08:41:14.261676+0000","last_became_active":"2024-03-21T08:41:12.339629+0000","last_became_peered":"2024-03-21T08:41:12.339629+0000","last_unstale":"2024-03-21T08:41:14.261676+0000","last_undegraded":"2024-03-21T08:41:14.261676+0000","last_fullsized":"2024-03-21T08:41:14.261676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:42.494428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161192+0000","last_change":"2024-03-21T08:41:12.340677+0000","last_active":"2024-03-21T08:41:13.161192+0000","last_peered":"2024-03-21T08:41:13.161192+0000","last_clean":"2024-03-21T08:41:13.161192+0000","last_became_active":"2024-03-21T08:41:12.340153+0000","last_became_peered":"2024-03-21T08:41:12.340153+0000","last_unstale":"2024-03-21T08:41:13.161192+0000","last_undegraded":"2024-03-21T08:41:13.161192+0000","last_fullsized":"2024-03-21T08:41:13.161192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:48.677726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261917+0000","last_change":"2024-03-21T08:41:08.004430+0000","last_active":"2024-03-21T08:41:14.261917+0000","last_peered":"2024-03-21T08:41:14.261917+0000","last_clean":"2024-03-21T08:41:14.261917+0000","last_became_active":"2024-03-21T08:41:08.003935+0000","last_became_peered":"2024-03-21T08:41:08.003935+0000","last_unstale":"2024-03-21T08:41:14.261917+0000","last_undegraded":"2024-03-21T08:41:14.261917+0000","last_fullsized":"2024-03-21T08:41:14.261917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260592+0000","last_change":"2024-03-21T08:41:07.011532+0000","last_active":"2024-03-21T08:41:14.260592+0000","last_peered":"2024-03-21T08:41:14.260592+0000","last_clean":"2024-03-21T08:41:14.260592+0000","last_became_active":"2024-03-21T08:41:07.008460+0000","last_became_peered":"2024-03-21T08:41:07.008460+0000","last_unstale":"2024-03-21T08:41:14.260592+0000","last_undegraded":"2024-03-21T08:41:14.260592+0000","last_fullsized":"2024-03-21T08:41:14.260592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026625+0000","last_change":"2024-03-21T08:41:12.054672+0000","last_active":"2024-03-21T08:41:13.026625+0000","last_peered":"2024-03-21T08:41:13.026625+0000","last_clean":"2024-03-21T08:41:13.026625+0000","last_became_active":"2024-03-21T08:41:12.054257+0000","last_became_peered":"2024-03-21T08:41:12.054257+0000","last_unstale":"2024-03-21T08:41:13.026625+0000","last_undegraded":"2024-03-21T08:41:13.026625+0000","last_fullsized":"2024-03-21T08:41:13.026625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261534+0000","last_change":"2024-03-21T08:41:08.015362+0000","last_active":"2024-03-21T08:41:14.261534+0000","last_peered":"2024-03-21T08:41:14.261534+0000","last_clean":"2024-03-21T08:41:14.261534+0000","last_became_active":"2024-03-21T08:41:08.015110+0000","last_became_peered":"2024-03-21T08:41:08.015110+0000","last_unstale":"2024-03-21T08:41:14.261534+0000","last_undegraded":"2024-03-21T08:41:14.261534+0000","last_fullsized":"2024-03-21T08:41:14.261534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261823+0000","last_change":"2024-03-21T08:41:07.001659+0000","last_active":"2024-03-21T08:41:14.261823+0000","last_peered":"2024-03-21T08:41:14.261823+0000","last_clean":"2024-03-21T08:41:14.261823+0000","last_became_active":"2024-03-21T08:41:07.001354+0000","last_became_peered":"2024-03-21T08:41:07.001354+0000","last_unstale":"2024-03-21T08:41:14.261823+0000","last_undegraded":"2024-03-21T08:41:14.261823+0000","last_fullsized":"2024-03-21T08:41:14.261823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.256597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026173+0000","last_change":"2024-03-21T08:41:07.009459+0000","last_active":"2024-03-21T08:41:13.026173+0000","last_peered":"2024-03-21T08:41:13.026173+0000","last_clean":"2024-03-21T08:41:13.026173+0000","last_became_active":"2024-03-21T08:41:07.009113+0000","last_became_peered":"2024-03-21T08:41:07.009113+0000","last_unstale":"2024-03-21T08:41:13.026173+0000","last_undegraded":"2024-03-21T08:41:13.026173+0000","last_fullsized":"2024-03-21T08:41:13.026173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161366+0000","last_change":"2024-03-21T08:41:07.996784+0000","last_active":"2024-03-21T08:41:13.161366+0000","last_peered":"2024-03-21T08:41:13.161366+0000","last_clean":"2024-03-21T08:41:13.161366+0000","last_became_active":"2024-03-21T08:41:07.996521+0000","last_became_peered":"2024-03-21T08:41:07.996521+0000","last_unstale":"2024-03-21T08:41:13.161366+0000","last_undegraded":"2024-03-21T08:41:13.161366+0000","last_fullsized":"2024-03-21T08:41:13.161366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025773+0000","last_change":"2024-03-21T08:41:12.052094+0000","last_active":"2024-03-21T08:41:13.025773+0000","last_peered":"2024-03-21T08:41:13.025773+0000","last_clean":"2024-03-21T08:41:13.025773+0000","last_became_active":"2024-03-21T08:41:12.051727+0000","last_became_peered":"2024-03-21T08:41:12.051727+0000","last_unstale":"2024-03-21T08:41:13.025773+0000","last_undegraded":"2024-03-21T08:41:13.025773+0000","last_fullsized":"2024-03-21T08:41:13.025773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:06:49.718295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262269+0000","last_change":"2024-03-21T08:41:07.012156+0000","last_active":"2024-03-21T08:41:14.262269+0000","last_peered":"2024-03-21T08:41:14.262269+0000","last_clean":"2024-03-21T08:41:14.262269+0000","last_became_active":"2024-03-21T08:41:07.011881+0000","last_became_peered":"2024-03-21T08:41:07.011881+0000","last_unstale":"2024-03-21T08:41:14.262269+0000","last_undegraded":"2024-03-21T08:41:14.262269+0000","last_fullsized":"2024-03-21T08:41:14.262269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027548+0000","last_change":"2024-03-21T08:41:08.002324+0000","last_active":"2024-03-21T08:41:13.027548+0000","last_peered":"2024-03-21T08:41:13.027548+0000","last_clean":"2024-03-21T08:41:13.027548+0000","last_became_active":"2024-03-21T08:41:08.002038+0000","last_became_peered":"2024-03-21T08:41:08.002038+0000","last_unstale":"2024-03-21T08:41:13.027548+0000","last_undegraded":"2024-03-21T08:41:13.027548+0000","last_fullsized":"2024-03-21T08:41:13.027548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:20.396047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262336+0000","last_change":"2024-03-21T08:41:12.058878+0000","last_active":"2024-03-21T08:41:14.262336+0000","last_peered":"2024-03-21T08:41:14.262336+0000","last_clean":"2024-03-21T08:41:14.262336+0000","last_became_active":"2024-03-21T08:41:12.058446+0000","last_became_peered":"2024-03-21T08:41:12.058446+0000","last_unstale":"2024-03-21T08:41:14.262336+0000","last_undegraded":"2024-03-21T08:41:14.262336+0000","last_fullsized":"2024-03-21T08:41:14.262336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:59:29.179384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025278+0000","last_change":"2024-03-21T08:41:08.010406+0000","last_active":"2024-03-21T08:41:13.025278+0000","last_peered":"2024-03-21T08:41:13.025278+0000","last_clean":"2024-03-21T08:41:13.025278+0000","last_became_active":"2024-03-21T08:41:08.009763+0000","last_became_peered":"2024-03-21T08:41:08.009763+0000","last_unstale":"2024-03-21T08:41:13.025278+0000","last_undegraded":"2024-03-21T08:41:13.025278+0000","last_fullsized":"2024-03-21T08:41:13.025278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025559+0000","last_change":"2024-03-21T08:41:08.009958+0000","last_active":"2024-03-21T08:41:13.025559+0000","last_peered":"2024-03-21T08:41:13.025559+0000","last_clean":"2024-03-21T08:41:13.025559+0000","last_became_active":"2024-03-21T08:41:08.009691+0000","last_became_peered":"2024-03-21T08:41:08.009691+0000","last_unstale":"2024-03-21T08:41:13.025559+0000","last_undegraded":"2024-03-21T08:41:13.025559+0000","last_fullsized":"2024-03-21T08:41:13.025559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:46.141146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260744+0000","last_change":"2024-03-21T08:41:08.004794+0000","last_active":"2024-03-21T08:41:14.260744+0000","last_peered":"2024-03-21T08:41:14.260744+0000","last_clean":"2024-03-21T08:41:14.260744+0000","last_became_active":"2024-03-21T08:41:08.003470+0000","last_became_peered":"2024-03-21T08:41:08.003470+0000","last_unstale":"2024-03-21T08:41:14.260744+0000","last_undegraded":"2024-03-21T08:41:14.260744+0000","last_fullsized":"2024-03-21T08:41:14.260744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161263+0000","last_change":"2024-03-21T08:41:08.014560+0000","last_active":"2024-03-21T08:41:13.161263+0000","last_peered":"2024-03-21T08:41:13.161263+0000","last_clean":"2024-03-21T08:41:13.161263+0000","last_became_active":"2024-03-21T08:41:08.014307+0000","last_became_peered":"2024-03-21T08:41:08.014307+0000","last_unstale":"2024-03-21T08:41:13.161263+0000","last_undegraded":"2024-03-21T08:41:13.161263+0000","last_fullsized":"2024-03-21T08:41:13.161263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262543+0000","last_change":"2024-03-21T08:41:08.026095+0000","last_active":"2024-03-21T08:41:14.262543+0000","last_peered":"2024-03-21T08:41:14.262543+0000","last_clean":"2024-03-21T08:41:14.262543+0000","last_became_active":"2024-03-21T08:41:08.025719+0000","last_became_peered":"2024-03-21T08:41:08.025719+0000","last_unstale":"2024-03-21T08:41:14.262543+0000","last_undegraded":"2024-03-21T08:41:14.262543+0000","last_fullsized":"2024-03-21T08:41:14.262543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262685+0000","last_change":"2024-03-21T08:41:08.007171+0000","last_active":"2024-03-21T08:41:14.262685+0000","last_peered":"2024-03-21T08:41:14.262685+0000","last_clean":"2024-03-21T08:41:14.262685+0000","last_became_active":"2024-03-21T08:41:08.006702+0000","last_became_peered":"2024-03-21T08:41:08.006702+0000","last_unstale":"2024-03-21T08:41:14.262685+0000","last_undegraded":"2024-03-21T08:41:14.262685+0000","last_fullsized":"2024-03-21T08:41:14.262685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026846+0000","last_change":"2024-03-21T08:41:08.029843+0000","last_active":"2024-03-21T08:41:13.026846+0000","last_peered":"2024-03-21T08:41:13.026846+0000","last_clean":"2024-03-21T08:41:13.026846+0000","last_became_active":"2024-03-21T08:41:08.029528+0000","last_became_peered":"2024-03-21T08:41:08.029528+0000","last_unstale":"2024-03-21T08:41:13.026846+0000","last_undegraded":"2024-03-21T08:41:13.026846+0000","last_fullsized":"2024-03-21T08:41:13.026846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262888+0000","last_change":"2024-03-21T08:41:08.013879+0000","last_active":"2024-03-21T08:41:14.262888+0000","last_peered":"2024-03-21T08:41:14.262888+0000","last_clean":"2024-03-21T08:41:14.262888+0000","last_became_active":"2024-03-21T08:41:08.013610+0000","last_became_peered":"2024-03-21T08:41:08.013610+0000","last_unstale":"2024-03-21T08:41:14.262888+0000","last_undegraded":"2024-03-21T08:41:14.262888+0000","last_fullsized":"2024-03-21T08:41:14.262888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260551+0000","last_change":"2024-03-21T08:41:08.001171+0000","last_active":"2024-03-21T08:41:14.260551+0000","last_peered":"2024-03-21T08:41:14.260551+0000","last_clean":"2024-03-21T08:41:14.260551+0000","last_became_active":"2024-03-21T08:41:08.000833+0000","last_became_peered":"2024-03-21T08:41:08.000833+0000","last_unstale":"2024-03-21T08:41:14.260551+0000","last_undegraded":"2024-03-21T08:41:14.260551+0000","last_fullsized":"2024-03-21T08:41:14.260551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261730+0000","last_change":"2024-03-21T08:41:08.014405+0000","last_active":"2024-03-21T08:41:14.261730+0000","last_peered":"2024-03-21T08:41:14.261730+0000","last_clean":"2024-03-21T08:41:14.261730+0000","last_became_active":"2024-03-21T08:41:08.014151+0000","last_became_peered":"2024-03-21T08:41:08.014151+0000","last_unstale":"2024-03-21T08:41:14.261730+0000","last_undegraded":"2024-03-21T08:41:14.261730+0000","last_fullsized":"2024-03-21T08:41:14.261730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260722+0000","last_change":"2024-03-21T08:41:08.019319+0000","last_active":"2024-03-21T08:41:14.260722+0000","last_peered":"2024-03-21T08:41:14.260722+0000","last_clean":"2024-03-21T08:41:14.260722+0000","last_became_active":"2024-03-21T08:41:08.019027+0000","last_became_peered":"2024-03-21T08:41:08.019027+0000","last_unstale":"2024-03-21T08:41:14.260722+0000","last_undegraded":"2024-03-21T08:41:14.260722+0000","last_fullsized":"2024-03-21T08:41:14.260722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161113+0000","last_change":"2024-03-21T08:41:08.018077+0000","last_active":"2024-03-21T08:41:13.161113+0000","last_peered":"2024-03-21T08:41:13.161113+0000","last_clean":"2024-03-21T08:41:13.161113+0000","last_became_active":"2024-03-21T08:41:08.017564+0000","last_became_peered":"2024-03-21T08:41:08.017564+0000","last_unstale":"2024-03-21T08:41:13.161113+0000","last_undegraded":"2024-03-21T08:41:13.161113+0000","last_fullsized":"2024-03-21T08:41:13.161113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026399+0000","last_change":"2024-03-21T08:41:08.018108+0000","last_active":"2024-03-21T08:41:13.026399+0000","last_peered":"2024-03-21T08:41:13.026399+0000","last_clean":"2024-03-21T08:41:13.026399+0000","last_became_active":"2024-03-21T08:41:08.017766+0000","last_became_peered":"2024-03-21T08:41:08.017766+0000","last_unstale":"2024-03-21T08:41:13.026399+0000","last_undegraded":"2024-03-21T08:41:13.026399+0000","last_fullsized":"2024-03-21T08:41:13.026399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737069+0000","last_change":"2024-03-21T08:41:08.021075+0000","last_active":"2024-03-21T08:41:13.737069+0000","last_peered":"2024-03-21T08:41:13.737069+0000","last_clean":"2024-03-21T08:41:13.737069+0000","last_became_active":"2024-03-21T08:41:08.020798+0000","last_became_peered":"2024-03-21T08:41:08.020798+0000","last_unstale":"2024-03-21T08:41:13.737069+0000","last_undegraded":"2024-03-21T08:41:13.737069+0000","last_fullsized":"2024-03-21T08:41:13.737069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:06.452855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025886+0000","last_change":"2024-03-21T08:41:08.008591+0000","last_active":"2024-03-21T08:41:13.025886+0000","last_peered":"2024-03-21T08:41:13.025886+0000","last_clean":"2024-03-21T08:41:13.025886+0000","last_became_active":"2024-03-21T08:41:08.008328+0000","last_became_peered":"2024-03-21T08:41:08.008328+0000","last_unstale":"2024-03-21T08:41:13.025886+0000","last_undegraded":"2024-03-21T08:41:13.025886+0000","last_fullsized":"2024-03-21T08:41:13.025886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:23:00.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261256+0000","last_change":"2024-03-21T08:41:08.023131+0000","last_active":"2024-03-21T08:41:14.261256+0000","last_peered":"2024-03-21T08:41:14.261256+0000","last_clean":"2024-03-21T08:41:14.261256+0000","last_became_active":"2024-03-21T08:41:08.022844+0000","last_became_peered":"2024-03-21T08:41:08.022844+0000","last_unstale":"2024-03-21T08:41:14.261256+0000","last_undegraded":"2024-03-21T08:41:14.261256+0000","last_fullsized":"2024-03-21T08:41:14.261256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:44.645908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260479+0000","last_change":"2024-03-21T08:41:08.008995+0000","last_active":"2024-03-21T08:41:14.260479+0000","last_peered":"2024-03-21T08:41:14.260479+0000","last_clean":"2024-03-21T08:41:14.260479+0000","last_became_active":"2024-03-21T08:41:08.008707+0000","last_became_peered":"2024-03-21T08:41:08.008707+0000","last_unstale":"2024-03-21T08:41:14.260479+0000","last_undegraded":"2024-03-21T08:41:14.260479+0000","last_fullsized":"2024-03-21T08:41:14.260479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261974+0000","last_change":"2024-03-21T08:41:08.007429+0000","last_active":"2024-03-21T08:41:14.261974+0000","last_peered":"2024-03-21T08:41:14.261974+0000","last_clean":"2024-03-21T08:41:14.261974+0000","last_became_active":"2024-03-21T08:41:08.007118+0000","last_became_peered":"2024-03-21T08:41:08.007118+0000","last_unstale":"2024-03-21T08:41:14.261974+0000","last_undegraded":"2024-03-21T08:41:14.261974+0000","last_fullsized":"2024-03-21T08:41:14.261974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160940+0000","last_change":"2024-03-21T08:41:08.002687+0000","last_active":"2024-03-21T08:41:13.160940+0000","last_peered":"2024-03-21T08:41:13.160940+0000","last_clean":"2024-03-21T08:41:13.160940+0000","last_became_active":"2024-03-21T08:41:08.002397+0000","last_became_peered":"2024-03-21T08:41:08.002397+0000","last_unstale":"2024-03-21T08:41:13.160940+0000","last_undegraded":"2024-03-21T08:41:13.160940+0000","last_fullsized":"2024-03-21T08:41:13.160940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:12:01.877634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262084+0000","last_change":"2024-03-21T08:41:08.011489+0000","last_active":"2024-03-21T08:41:14.262084+0000","last_peered":"2024-03-21T08:41:14.262084+0000","last_clean":"2024-03-21T08:41:14.262084+0000","last_became_active":"2024-03-21T08:41:08.011227+0000","last_became_peered":"2024-03-21T08:41:08.011227+0000","last_unstale":"2024-03-21T08:41:14.262084+0000","last_undegraded":"2024-03-21T08:41:14.262084+0000","last_fullsized":"2024-03-21T08:41:14.262084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027108+0000","last_change":"2024-03-21T08:41:08.010854+0000","last_active":"2024-03-21T08:41:13.027108+0000","last_peered":"2024-03-21T08:41:13.027108+0000","last_clean":"2024-03-21T08:41:13.027108+0000","last_became_active":"2024-03-21T08:41:08.010557+0000","last_became_peered":"2024-03-21T08:41:08.010557+0000","last_unstale":"2024-03-21T08:41:13.027108+0000","last_undegraded":"2024-03-21T08:41:13.027108+0000","last_fullsized":"2024-03-21T08:41:13.027108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:01.730277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026506+0000","last_change":"2024-03-21T08:41:08.014812+0000","last_active":"2024-03-21T08:41:13.026506+0000","last_peered":"2024-03-21T08:41:13.026506+0000","last_clean":"2024-03-21T08:41:13.026506+0000","last_became_active":"2024-03-21T08:41:08.014508+0000","last_became_peered":"2024-03-21T08:41:08.014508+0000","last_unstale":"2024-03-21T08:41:13.026506+0000","last_undegraded":"2024-03-21T08:41:13.026506+0000","last_fullsized":"2024-03-21T08:41:13.026506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026766+0000","last_change":"2024-03-21T08:41:08.013816+0000","last_active":"2024-03-21T08:41:13.026766+0000","last_peered":"2024-03-21T08:41:13.026766+0000","last_clean":"2024-03-21T08:41:13.026766+0000","last_became_active":"2024-03-21T08:41:08.013585+0000","last_became_peered":"2024-03-21T08:41:08.013585+0000","last_unstale":"2024-03-21T08:41:13.026766+0000","last_undegraded":"2024-03-21T08:41:13.026766+0000","last_fullsized":"2024-03-21T08:41:13.026766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737247+0000","last_change":"2024-03-21T08:41:08.031989+0000","last_active":"2024-03-21T08:41:13.737247+0000","last_peered":"2024-03-21T08:41:13.737247+0000","last_clean":"2024-03-21T08:41:13.737247+0000","last_became_active":"2024-03-21T08:41:08.031755+0000","last_became_peered":"2024-03-21T08:41:08.031755+0000","last_unstale":"2024-03-21T08:41:13.737247+0000","last_undegraded":"2024-03-21T08:41:13.737247+0000","last_fullsized":"2024-03-21T08:41:13.737247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:37.172205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261442+0000","last_change":"2024-03-21T08:41:08.015070+0000","last_active":"2024-03-21T08:41:14.261442+0000","last_peered":"2024-03-21T08:41:14.261442+0000","last_clean":"2024-03-21T08:41:14.261442+0000","last_became_active":"2024-03-21T08:41:08.012083+0000","last_became_peered":"2024-03-21T08:41:08.012083+0000","last_unstale":"2024-03-21T08:41:14.261442+0000","last_undegraded":"2024-03-21T08:41:14.261442+0000","last_fullsized":"2024-03-21T08:41:14.261442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736535+0000","last_change":"2024-03-21T08:41:08.020740+0000","last_active":"2024-03-21T08:41:13.736535+0000","last_peered":"2024-03-21T08:41:13.736535+0000","last_clean":"2024-03-21T08:41:13.736535+0000","last_became_active":"2024-03-21T08:41:08.020396+0000","last_became_peered":"2024-03-21T08:41:08.020396+0000","last_unstale":"2024-03-21T08:41:13.736535+0000","last_undegraded":"2024-03-21T08:41:13.736535+0000","last_fullsized":"2024-03-21T08:41:13.736535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:59:37.815273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260654+0000","last_change":"2024-03-21T08:41:08.010314+0000","last_active":"2024-03-21T08:41:14.260654+0000","last_peered":"2024-03-21T08:41:14.260654+0000","last_clean":"2024-03-21T08:41:14.260654+0000","last_became_active":"2024-03-21T08:41:08.009693+0000","last_became_peered":"2024-03-21T08:41:08.009693+0000","last_unstale":"2024-03-21T08:41:14.260654+0000","last_undegraded":"2024-03-21T08:41:14.260654+0000","last_fullsized":"2024-03-21T08:41:14.260654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736584+0000","last_change":"2024-03-21T08:41:08.020466+0000","last_active":"2024-03-21T08:41:13.736584+0000","last_peered":"2024-03-21T08:41:13.736584+0000","last_clean":"2024-03-21T08:41:13.736584+0000","last_became_active":"2024-03-21T08:41:08.020169+0000","last_became_peered":"2024-03-21T08:41:08.020169+0000","last_unstale":"2024-03-21T08:41:13.736584+0000","last_undegraded":"2024-03-21T08:41:13.736584+0000","last_fullsized":"2024-03-21T08:41:13.736584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:00.508480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260900+0000","last_change":"2024-03-21T08:41:08.014151+0000","last_active":"2024-03-21T08:41:14.260900+0000","last_peered":"2024-03-21T08:41:14.260900+0000","last_clean":"2024-03-21T08:41:14.260900+0000","last_became_active":"2024-03-21T08:41:08.013319+0000","last_became_peered":"2024-03-21T08:41:08.013319+0000","last_unstale":"2024-03-21T08:41:14.260900+0000","last_undegraded":"2024-03-21T08:41:14.260900+0000","last_fullsized":"2024-03-21T08:41:14.260900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:16.725366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161103+0000","last_change":"2024-03-21T08:41:08.002931+0000","last_active":"2024-03-21T08:41:13.161103+0000","last_peered":"2024-03-21T08:41:13.161103+0000","last_clean":"2024-03-21T08:41:13.161103+0000","last_became_active":"2024-03-21T08:41:08.001953+0000","last_became_peered":"2024-03-21T08:41:08.001953+0000","last_unstale":"2024-03-21T08:41:13.161103+0000","last_undegraded":"2024-03-21T08:41:13.161103+0000","last_fullsized":"2024-03-21T08:41:13.161103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263385+0000","last_change":"2024-03-21T08:41:08.005883+0000","last_active":"2024-03-21T08:41:14.263385+0000","last_peered":"2024-03-21T08:41:14.263385+0000","last_clean":"2024-03-21T08:41:14.263385+0000","last_became_active":"2024-03-21T08:41:08.005503+0000","last_became_peered":"2024-03-21T08:41:08.005503+0000","last_unstale":"2024-03-21T08:41:14.263385+0000","last_undegraded":"2024-03-21T08:41:14.263385+0000","last_fullsized":"2024-03-21T08:41:14.263385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737138+0000","last_change":"2024-03-21T08:41:08.030286+0000","last_active":"2024-03-21T08:41:13.737138+0000","last_peered":"2024-03-21T08:41:13.737138+0000","last_clean":"2024-03-21T08:41:13.737138+0000","last_became_active":"2024-03-21T08:41:08.029830+0000","last_became_peered":"2024-03-21T08:41:08.029830+0000","last_unstale":"2024-03-21T08:41:13.737138+0000","last_undegraded":"2024-03-21T08:41:13.737138+0000","last_fullsized":"2024-03-21T08:41:13.737138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:31.510333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":693167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":102377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":693223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:41:27.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:41:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.362+0000 7f13e8200640 1 Processor -- start 2024-03-21T08:41:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.362+0000 7f13e8200640 1 -- start start 2024-03-21T08:41:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e8200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 0x7f13e015f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e8200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00cfac0 con 0x7f13e00d0880 2024-03-21T08:41:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e8200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00cfda0 con 0x7f13e0099b80 2024-03-21T08:41:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e8200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00d0080 con 0x7f13e00b0b40 2024-03-21T08:41:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f13e00d0880 0x7f13e006c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36788/0 (socket says 172.21.15.28:36788) 2024-03-21T08:41:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e4e00640 1 -- 172.21.15.28:0/3136854721 learned_addr learned my addr 172.21.15.28:0/3136854721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e6200640 1 --2- 172.21.15.28:0/3136854721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 0x7f13e015f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208003674 0 0) 0x7f13e00cfac0 con 0x7f13e00d0880 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003530 con 0x7f13e00d0880 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563887095 0 0) 0x7f13e00cfda0 con 0x7f13e0099b80 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e00cfac0 con 0x7f13e0099b80 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e6200640 1 -- 172.21.15.28:0/3136854721 >> v1:172.21.15.151:6789/0 conn(0x7f13e0099b80 legacy=0x7f13e0099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e6200640 1 -- 172.21.15.28:0/3136854721 >> v1:172.21.15.28:6789/0 conn(0x7f13e00d0880 legacy=0x7f13e006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.366+0000 7f13e6200640 1 -- 172.21.15.28:0/3136854721 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00d0360 con 0x7f13e00b0b40 2024-03-21T08:41:27.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e6200640 1 --2- 172.21.15.28:0/3136854721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 0x7f13e015f220 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009dc0 tx=0x7f13d801c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab6e6e6d108576c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d80b2070 con 0x7f13e00b0b40 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d80a5970 con 0x7f13e00b0b40 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13dfe00640 1 -- 172.21.15.28:0/3136854721 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d80a5d40 con 0x7f13e00b0b40 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- 172.21.15.28:0/3136854721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 msgr2=0x7f13e015f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 --2- 172.21.15.28:0/3136854721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 0x7f13e015f220 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009dc0 tx=0x7f13d801c990 comp rx=0 tx=0).stop 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- 172.21.15.28:0/3136854721 shutdown_connections 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 --2- 172.21.15.28:0/3136854721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00b0b40 0x7f13e015f220 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- 172.21.15.28:0/3136854721 >> 172.21.15.28:0/3136854721 conn(0x7f13e00b3510 msgr2=0x7f13e00b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- 172.21.15.28:0/3136854721 shutdown_connections 2024-03-21T08:41:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- 172.21.15.28:0/3136854721 wait complete. 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 Processor -- start 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.370+0000 7f13e8200640 1 -- start start 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13e0099b80 0x7f13e009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00d0880 0x7f13e00c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e016c3f0 con 0x7f13e00b0b40 2024-03-21T08:41:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e016c270 con 0x7f13e0099b80 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e016c570 con 0x7f13e00d0880 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51386/0 (socket says 172.21.15.28:51386) 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 -- 172.21.15.28:0/3353814455 learned_addr learned my addr 172.21.15.28:0/3353814455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e5800640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13e0099b80 0x7f13e009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e6200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00d0880 0x7f13e00c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 -- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00d0880 msgr2=0x7f13e00c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00d0880 0x7f13e00c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 -- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13e0099b80 msgr2=0x7f13e009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13e0099b80 0x7f13e009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 -- 172.21.15.28:0/3353814455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e009e360 con 0x7f13e00b0b40 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e4e00640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c420 tx=0x7f13cc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc017070 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc005cf0 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0072d0 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e009e640 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e009ec00 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e01537f0 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.374+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f13cc01a070 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.378+0000 7f13dd600640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 0x7f13c006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.378+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f13cc0854e0 con 0x7f13e00b0b40 2024-03-21T08:41:27.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.378+0000 7f13e5800640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 0x7f13c006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.378+0000 7f13e5800640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 0x7f13c006a010 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13e015ad80 tx=0x7f13d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.382+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f13e01537f0 con 0x7f13e00b0b40 2024-03-21T08:41:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.546+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13e004f410 con 0x7f13c0067b50 2024-03-21T08:41:27.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13dd600640 1 -- 172.21.15.28:0/3353814455 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332562 (secure 0 0 0) 0x7f13e004f410 con 0x7f13c0067b50 2024-03-21T08:41:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 msgr2=0x7f13c006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 0x7f13c006a010 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f13e015ad80 tx=0x7f13d4009040 comp rx=0 tx=0).stop 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 msgr2=0x7f13e009d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00c420 tx=0x7f13cc00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 shutdown_connections 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13c0067b50 0x7f13c006a010 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f13e00d0880 0x7f13e00c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e00b0b40 0x7f13e009d8f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 --2- 172.21.15.28:0/3353814455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f13e0099b80 0x7f13e009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 >> 172.21.15.28:0/3353814455 conn(0x7f13e00b3510 msgr2=0x7f13e00b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 shutdown_connections 2024-03-21T08:41:27.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.566+0000 7f13e8200640 1 -- 172.21.15.28:0/3353814455 wait complete. 2024-03-21T08:41:27.590 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-21T08:41:25.722503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231892,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742828,"statfs":{"total":773094113280,"available":772856655872,"internally_reserved":0,"allocated":3317760,"data_stored":2007752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161451+0000","last_change":"2024-03-21T08:41:07.021438+0000","last_active":"2024-03-21T08:41:13.161451+0000","last_peered":"2024-03-21T08:41:13.161451+0000","last_clean":"2024-03-21T08:41:13.161451+0000","last_became_active":"2024-03-21T08:41:07.021142+0000","last_became_peered":"2024-03-21T08:41:07.021142+0000","last_unstale":"2024-03-21T08:41:13.161451+0000","last_undegraded":"2024-03-21T08:41:13.161451+0000","last_fullsized":"2024-03-21T08:41:13.161451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:15:32.936778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262630+0000","last_change":"2024-03-21T08:41:08.010424+0000","last_active":"2024-03-21T08:41:14.262630+0000","last_peered":"2024-03-21T08:41:14.262630+0000","last_clean":"2024-03-21T08:41:14.262630+0000","last_became_active":"2024-03-21T08:41:08.010169+0000","last_became_peered":"2024-03-21T08:41:08.010169+0000","last_unstale":"2024-03-21T08:41:14.262630+0000","last_undegraded":"2024-03-21T08:41:14.262630+0000","last_fullsized":"2024-03-21T08:41:14.262630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261510+0000","last_change":"2024-03-21T08:41:12.340891+0000","last_active":"2024-03-21T08:41:14.261510+0000","last_peered":"2024-03-21T08:41:14.261510+0000","last_clean":"2024-03-21T08:41:14.261510+0000","last_became_active":"2024-03-21T08:41:12.340181+0000","last_became_peered":"2024-03-21T08:41:12.340181+0000","last_unstale":"2024-03-21T08:41:14.261510+0000","last_undegraded":"2024-03-21T08:41:14.261510+0000","last_fullsized":"2024-03-21T08:41:14.261510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262757+0000","last_change":"2024-03-21T08:41:07.009640+0000","last_active":"2024-03-21T08:41:14.262757+0000","last_peered":"2024-03-21T08:41:14.262757+0000","last_clean":"2024-03-21T08:41:14.262757+0000","last_became_active":"2024-03-21T08:41:07.009390+0000","last_became_peered":"2024-03-21T08:41:07.009390+0000","last_unstale":"2024-03-21T08:41:14.262757+0000","last_undegraded":"2024-03-21T08:41:14.262757+0000","last_fullsized":"2024-03-21T08:41:14.262757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161250+0000","last_change":"2024-03-21T08:41:08.010008+0000","last_active":"2024-03-21T08:41:13.161250+0000","last_peered":"2024-03-21T08:41:13.161250+0000","last_clean":"2024-03-21T08:41:13.161250+0000","last_became_active":"2024-03-21T08:41:08.008163+0000","last_became_peered":"2024-03-21T08:41:08.008163+0000","last_unstale":"2024-03-21T08:41:13.161250+0000","last_undegraded":"2024-03-21T08:41:13.161250+0000","last_fullsized":"2024-03-21T08:41:13.161250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025449+0000","last_change":"2024-03-21T08:41:12.049570+0000","last_active":"2024-03-21T08:41:13.025449+0000","last_peered":"2024-03-21T08:41:13.025449+0000","last_clean":"2024-03-21T08:41:13.025449+0000","last_became_active":"2024-03-21T08:41:12.048903+0000","last_became_peered":"2024-03-21T08:41:12.048903+0000","last_unstale":"2024-03-21T08:41:13.025449+0000","last_undegraded":"2024-03-21T08:41:13.025449+0000","last_fullsized":"2024-03-21T08:41:13.025449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:10.935038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736993+0000","last_change":"2024-03-21T08:41:07.022988+0000","last_active":"2024-03-21T08:41:13.736993+0000","last_peered":"2024-03-21T08:41:13.736993+0000","last_clean":"2024-03-21T08:41:13.736993+0000","last_became_active":"2024-03-21T08:41:07.022643+0000","last_became_peered":"2024-03-21T08:41:07.022643+0000","last_unstale":"2024-03-21T08:41:13.736993+0000","last_undegraded":"2024-03-21T08:41:13.736993+0000","last_fullsized":"2024-03-21T08:41:13.736993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260935+0000","last_change":"2024-03-21T08:41:08.008250+0000","last_active":"2024-03-21T08:41:14.260935+0000","last_peered":"2024-03-21T08:41:14.260935+0000","last_clean":"2024-03-21T08:41:14.260935+0000","last_became_active":"2024-03-21T08:41:08.007638+0000","last_became_peered":"2024-03-21T08:41:08.007638+0000","last_unstale":"2024-03-21T08:41:14.260935+0000","last_undegraded":"2024-03-21T08:41:14.260935+0000","last_fullsized":"2024-03-21T08:41:14.260935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:14.189872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027230+0000","last_change":"2024-03-21T08:41:12.051136+0000","last_active":"2024-03-21T08:41:13.027230+0000","last_peered":"2024-03-21T08:41:13.027230+0000","last_clean":"2024-03-21T08:41:13.027230+0000","last_became_active":"2024-03-21T08:41:12.050706+0000","last_became_peered":"2024-03-21T08:41:12.050706+0000","last_unstale":"2024-03-21T08:41:13.027230+0000","last_undegraded":"2024-03-21T08:41:13.027230+0000","last_fullsized":"2024-03-21T08:41:13.027230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:28:03.320473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736865+0000","last_change":"2024-03-21T08:41:07.021316+0000","last_active":"2024-03-21T08:41:13.736865+0000","last_peered":"2024-03-21T08:41:13.736865+0000","last_clean":"2024-03-21T08:41:13.736865+0000","last_became_active":"2024-03-21T08:41:07.021039+0000","last_became_peered":"2024-03-21T08:41:07.021039+0000","last_unstale":"2024-03-21T08:41:13.736865+0000","last_undegraded":"2024-03-21T08:41:13.736865+0000","last_fullsized":"2024-03-21T08:41:13.736865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261060+0000","last_change":"2024-03-21T08:41:08.004273+0000","last_active":"2024-03-21T08:41:14.261060+0000","last_peered":"2024-03-21T08:41:14.261060+0000","last_clean":"2024-03-21T08:41:14.261060+0000","last_became_active":"2024-03-21T08:41:08.003800+0000","last_became_peered":"2024-03-21T08:41:08.003800+0000","last_unstale":"2024-03-21T08:41:14.261060+0000","last_undegraded":"2024-03-21T08:41:14.261060+0000","last_fullsized":"2024-03-21T08:41:14.261060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027771+0000","last_change":"2024-03-21T08:41:12.046634+0000","last_active":"2024-03-21T08:41:13.027771+0000","last_peered":"2024-03-21T08:41:13.027771+0000","last_clean":"2024-03-21T08:41:13.027771+0000","last_became_active":"2024-03-21T08:41:12.046224+0000","last_became_peered":"2024-03-21T08:41:12.046224+0000","last_unstale":"2024-03-21T08:41:13.027771+0000","last_undegraded":"2024-03-21T08:41:13.027771+0000","last_fullsized":"2024-03-21T08:41:13.027771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:20.757184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025520+0000","last_change":"2024-03-21T08:41:08.009427+0000","last_active":"2024-03-21T08:41:13.025520+0000","last_peered":"2024-03-21T08:41:13.025520+0000","last_clean":"2024-03-21T08:41:13.025520+0000","last_became_active":"2024-03-21T08:41:08.009082+0000","last_became_peered":"2024-03-21T08:41:08.009082+0000","last_unstale":"2024-03-21T08:41:13.025520+0000","last_undegraded":"2024-03-21T08:41:13.025520+0000","last_fullsized":"2024-03-21T08:41:13.025520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:07.174703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736774+0000","last_change":"2024-03-21T08:41:07.019968+0000","last_active":"2024-03-21T08:41:13.736774+0000","last_peered":"2024-03-21T08:41:13.736774+0000","last_clean":"2024-03-21T08:41:13.736774+0000","last_became_active":"2024-03-21T08:41:07.019039+0000","last_became_peered":"2024-03-21T08:41:07.019039+0000","last_unstale":"2024-03-21T08:41:13.736774+0000","last_undegraded":"2024-03-21T08:41:13.736774+0000","last_fullsized":"2024-03-21T08:41:13.736774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027696+0000","last_change":"2024-03-21T08:41:12.053357+0000","last_active":"2024-03-21T08:41:13.027696+0000","last_peered":"2024-03-21T08:41:13.027696+0000","last_clean":"2024-03-21T08:41:13.027696+0000","last_became_active":"2024-03-21T08:41:12.053045+0000","last_became_peered":"2024-03-21T08:41:12.053045+0000","last_unstale":"2024-03-21T08:41:13.027696+0000","last_undegraded":"2024-03-21T08:41:13.027696+0000","last_fullsized":"2024-03-21T08:41:13.027696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:52.060002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262174+0000","last_change":"2024-03-21T08:41:08.014032+0000","last_active":"2024-03-21T08:41:14.262174+0000","last_peered":"2024-03-21T08:41:14.262174+0000","last_clean":"2024-03-21T08:41:14.262174+0000","last_became_active":"2024-03-21T08:41:08.013716+0000","last_became_peered":"2024-03-21T08:41:08.013716+0000","last_unstale":"2024-03-21T08:41:14.262174+0000","last_undegraded":"2024-03-21T08:41:14.262174+0000","last_fullsized":"2024-03-21T08:41:14.262174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262296+0000","last_change":"2024-03-21T08:41:07.008347+0000","last_active":"2024-03-21T08:41:14.262296+0000","last_peered":"2024-03-21T08:41:14.262296+0000","last_clean":"2024-03-21T08:41:14.262296+0000","last_became_active":"2024-03-21T08:41:07.005441+0000","last_became_peered":"2024-03-21T08:41:07.005441+0000","last_unstale":"2024-03-21T08:41:14.262296+0000","last_undegraded":"2024-03-21T08:41:14.262296+0000","last_fullsized":"2024-03-21T08:41:14.262296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025374+0000","last_change":"2024-03-21T08:41:12.051147+0000","last_active":"2024-03-21T08:41:13.025374+0000","last_peered":"2024-03-21T08:41:13.025374+0000","last_clean":"2024-03-21T08:41:13.025374+0000","last_became_active":"2024-03-21T08:41:12.050744+0000","last_became_peered":"2024-03-21T08:41:12.050744+0000","last_unstale":"2024-03-21T08:41:13.025374+0000","last_undegraded":"2024-03-21T08:41:13.025374+0000","last_fullsized":"2024-03-21T08:41:13.025374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:15.657762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261625+0000","last_change":"2024-03-21T08:41:08.003261+0000","last_active":"2024-03-21T08:41:14.261625+0000","last_peered":"2024-03-21T08:41:14.261625+0000","last_clean":"2024-03-21T08:41:14.261625+0000","last_became_active":"2024-03-21T08:41:08.002934+0000","last_became_peered":"2024-03-21T08:41:08.002934+0000","last_unstale":"2024-03-21T08:41:14.261625+0000","last_undegraded":"2024-03-21T08:41:14.261625+0000","last_fullsized":"2024-03-21T08:41:14.261625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262016+0000","last_change":"2024-03-21T08:41:07.020788+0000","last_active":"2024-03-21T08:41:14.262016+0000","last_peered":"2024-03-21T08:41:14.262016+0000","last_clean":"2024-03-21T08:41:14.262016+0000","last_became_active":"2024-03-21T08:41:07.020341+0000","last_became_peered":"2024-03-21T08:41:07.020341+0000","last_unstale":"2024-03-21T08:41:14.262016+0000","last_undegraded":"2024-03-21T08:41:14.262016+0000","last_fullsized":"2024-03-21T08:41:14.262016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263514+0000","last_change":"2024-03-21T08:41:12.340482+0000","last_active":"2024-03-21T08:41:14.263514+0000","last_peered":"2024-03-21T08:41:14.263514+0000","last_clean":"2024-03-21T08:41:14.263514+0000","last_became_active":"2024-03-21T08:41:12.340076+0000","last_became_peered":"2024-03-21T08:41:12.340076+0000","last_unstale":"2024-03-21T08:41:14.263514+0000","last_undegraded":"2024-03-21T08:41:14.263514+0000","last_fullsized":"2024-03-21T08:41:14.263514+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:10.484672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027247+0000","last_change":"2024-03-21T08:41:08.002403+0000","last_active":"2024-03-21T08:41:13.027247+0000","last_peered":"2024-03-21T08:41:13.027247+0000","last_clean":"2024-03-21T08:41:13.027247+0000","last_became_active":"2024-03-21T08:41:08.001933+0000","last_became_peered":"2024-03-21T08:41:08.001933+0000","last_unstale":"2024-03-21T08:41:13.027247+0000","last_undegraded":"2024-03-21T08:41:13.027247+0000","last_fullsized":"2024-03-21T08:41:13.027247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:40.121158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736405+0000","last_change":"2024-03-21T08:41:07.019255+0000","last_active":"2024-03-21T08:41:13.736405+0000","last_peered":"2024-03-21T08:41:13.736405+0000","last_clean":"2024-03-21T08:41:13.736405+0000","last_became_active":"2024-03-21T08:41:07.018958+0000","last_became_peered":"2024-03-21T08:41:07.018958+0000","last_unstale":"2024-03-21T08:41:13.736405+0000","last_undegraded":"2024-03-21T08:41:13.736405+0000","last_fullsized":"2024-03-21T08:41:13.736405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027872+0000","last_change":"2024-03-21T08:41:12.049104+0000","last_active":"2024-03-21T08:41:13.027872+0000","last_peered":"2024-03-21T08:41:13.027872+0000","last_clean":"2024-03-21T08:41:13.027872+0000","last_became_active":"2024-03-21T08:41:12.048724+0000","last_became_peered":"2024-03-21T08:41:12.048724+0000","last_unstale":"2024-03-21T08:41:13.027872+0000","last_undegraded":"2024-03-21T08:41:13.027872+0000","last_fullsized":"2024-03-21T08:41:13.027872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161500+0000","last_change":"2024-03-21T08:41:08.006976+0000","last_active":"2024-03-21T08:41:13.161500+0000","last_peered":"2024-03-21T08:41:13.161500+0000","last_clean":"2024-03-21T08:41:13.161500+0000","last_became_active":"2024-03-21T08:41:08.006565+0000","last_became_peered":"2024-03-21T08:41:08.006565+0000","last_unstale":"2024-03-21T08:41:13.161500+0000","last_undegraded":"2024-03-21T08:41:13.161500+0000","last_fullsized":"2024-03-21T08:41:13.161500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026772+0000","last_change":"2024-03-21T08:41:07.014131+0000","last_active":"2024-03-21T08:41:13.026772+0000","last_peered":"2024-03-21T08:41:13.026772+0000","last_clean":"2024-03-21T08:41:13.026772+0000","last_became_active":"2024-03-21T08:41:07.013789+0000","last_became_peered":"2024-03-21T08:41:07.013789+0000","last_unstale":"2024-03-21T08:41:13.026772+0000","last_undegraded":"2024-03-21T08:41:13.026772+0000","last_fullsized":"2024-03-21T08:41:13.026772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027502+0000","last_change":"2024-03-21T08:41:12.336087+0000","last_active":"2024-03-21T08:41:13.027502+0000","last_peered":"2024-03-21T08:41:13.027502+0000","last_clean":"2024-03-21T08:41:13.027502+0000","last_became_active":"2024-03-21T08:41:12.335722+0000","last_became_peered":"2024-03-21T08:41:12.335722+0000","last_unstale":"2024-03-21T08:41:13.027502+0000","last_undegraded":"2024-03-21T08:41:13.027502+0000","last_fullsized":"2024-03-21T08:41:13.027502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:47.106105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262994+0000","last_change":"2024-03-21T08:41:08.021861+0000","last_active":"2024-03-21T08:41:14.262994+0000","last_peered":"2024-03-21T08:41:14.262994+0000","last_clean":"2024-03-21T08:41:14.262994+0000","last_became_active":"2024-03-21T08:41:08.021564+0000","last_became_peered":"2024-03-21T08:41:08.021564+0000","last_unstale":"2024-03-21T08:41:14.262994+0000","last_undegraded":"2024-03-21T08:41:14.262994+0000","last_fullsized":"2024-03-21T08:41:14.262994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263418+0000","last_change":"2024-03-21T08:41:07.010549+0000","last_active":"2024-03-21T08:41:14.263418+0000","last_peered":"2024-03-21T08:41:14.263418+0000","last_clean":"2024-03-21T08:41:14.263418+0000","last_became_active":"2024-03-21T08:41:07.010129+0000","last_became_peered":"2024-03-21T08:41:07.010129+0000","last_unstale":"2024-03-21T08:41:14.263418+0000","last_undegraded":"2024-03-21T08:41:14.263418+0000","last_fullsized":"2024-03-21T08:41:14.263418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261368+0000","last_change":"2024-03-21T08:41:12.340111+0000","last_active":"2024-03-21T08:41:14.261368+0000","last_peered":"2024-03-21T08:41:14.261368+0000","last_clean":"2024-03-21T08:41:14.261368+0000","last_became_active":"2024-03-21T08:41:12.339714+0000","last_became_peered":"2024-03-21T08:41:12.339714+0000","last_unstale":"2024-03-21T08:41:14.261368+0000","last_undegraded":"2024-03-21T08:41:14.261368+0000","last_fullsized":"2024-03-21T08:41:14.261368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:30:09.350945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262982+0000","last_change":"2024-03-21T08:41:08.005617+0000","last_active":"2024-03-21T08:41:14.262982+0000","last_peered":"2024-03-21T08:41:14.262982+0000","last_clean":"2024-03-21T08:41:14.262982+0000","last_became_active":"2024-03-21T08:41:08.005254+0000","last_became_peered":"2024-03-21T08:41:08.005254+0000","last_unstale":"2024-03-21T08:41:14.262982+0000","last_undegraded":"2024-03-21T08:41:14.262982+0000","last_fullsized":"2024-03-21T08:41:14.262982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737093+0000","last_change":"2024-03-21T08:41:07.019857+0000","last_active":"2024-03-21T08:41:13.737093+0000","last_peered":"2024-03-21T08:41:13.737093+0000","last_clean":"2024-03-21T08:41:13.737093+0000","last_became_active":"2024-03-21T08:41:07.019553+0000","last_became_peered":"2024-03-21T08:41:07.019553+0000","last_unstale":"2024-03-21T08:41:13.737093+0000","last_undegraded":"2024-03-21T08:41:13.737093+0000","last_fullsized":"2024-03-21T08:41:13.737093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:35:34.563261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161134+0000","last_change":"2024-03-21T08:41:12.335655+0000","last_active":"2024-03-21T08:41:13.161134+0000","last_peered":"2024-03-21T08:41:13.161134+0000","last_clean":"2024-03-21T08:41:13.161134+0000","last_became_active":"2024-03-21T08:41:12.335219+0000","last_became_peered":"2024-03-21T08:41:12.335219+0000","last_unstale":"2024-03-21T08:41:13.161134+0000","last_undegraded":"2024-03-21T08:41:13.161134+0000","last_fullsized":"2024-03-21T08:41:13.161134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261158+0000","last_change":"2024-03-21T08:41:08.016121+0000","last_active":"2024-03-21T08:41:14.261158+0000","last_peered":"2024-03-21T08:41:14.261158+0000","last_clean":"2024-03-21T08:41:14.261158+0000","last_became_active":"2024-03-21T08:41:08.015237+0000","last_became_peered":"2024-03-21T08:41:08.015237+0000","last_unstale":"2024-03-21T08:41:14.261158+0000","last_undegraded":"2024-03-21T08:41:14.261158+0000","last_fullsized":"2024-03-21T08:41:14.261158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:48.028184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263069+0000","last_change":"2024-03-21T08:41:07.013927+0000","last_active":"2024-03-21T08:41:14.263069+0000","last_peered":"2024-03-21T08:41:14.263069+0000","last_clean":"2024-03-21T08:41:14.263069+0000","last_became_active":"2024-03-21T08:41:07.013595+0000","last_became_peered":"2024-03-21T08:41:07.013595+0000","last_unstale":"2024-03-21T08:41:14.263069+0000","last_undegraded":"2024-03-21T08:41:14.263069+0000","last_fullsized":"2024-03-21T08:41:14.263069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262181+0000","last_change":"2024-03-21T08:41:12.056397+0000","last_active":"2024-03-21T08:41:14.262181+0000","last_peered":"2024-03-21T08:41:14.262181+0000","last_clean":"2024-03-21T08:41:14.262181+0000","last_became_active":"2024-03-21T08:41:12.054349+0000","last_became_peered":"2024-03-21T08:41:12.054349+0000","last_unstale":"2024-03-21T08:41:14.262181+0000","last_undegraded":"2024-03-21T08:41:14.262181+0000","last_fullsized":"2024-03-21T08:41:14.262181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027232+0000","last_change":"2024-03-21T08:41:07.997069+0000","last_active":"2024-03-21T08:41:13.027232+0000","last_peered":"2024-03-21T08:41:13.027232+0000","last_clean":"2024-03-21T08:41:13.027232+0000","last_became_active":"2024-03-21T08:41:07.996691+0000","last_became_peered":"2024-03-21T08:41:07.996691+0000","last_unstale":"2024-03-21T08:41:13.027232+0000","last_undegraded":"2024-03-21T08:41:13.027232+0000","last_fullsized":"2024-03-21T08:41:13.027232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:59.860172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260862+0000","last_change":"2024-03-21T08:41:07.004740+0000","last_active":"2024-03-21T08:41:14.260862+0000","last_peered":"2024-03-21T08:41:14.260862+0000","last_clean":"2024-03-21T08:41:14.260862+0000","last_became_active":"2024-03-21T08:41:07.004315+0000","last_became_peered":"2024-03-21T08:41:07.004315+0000","last_unstale":"2024-03-21T08:41:14.260862+0000","last_undegraded":"2024-03-21T08:41:14.260862+0000","last_fullsized":"2024-03-21T08:41:14.260862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:58.854457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263063+0000","last_change":"2024-03-21T08:41:12.049583+0000","last_active":"2024-03-21T08:41:14.263063+0000","last_peered":"2024-03-21T08:41:14.263063+0000","last_clean":"2024-03-21T08:41:14.263063+0000","last_became_active":"2024-03-21T08:41:12.048879+0000","last_became_peered":"2024-03-21T08:41:12.048879+0000","last_unstale":"2024-03-21T08:41:14.263063+0000","last_undegraded":"2024-03-21T08:41:14.263063+0000","last_fullsized":"2024-03-21T08:41:14.263063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:41.204023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261754+0000","last_change":"2024-03-21T08:41:08.006302+0000","last_active":"2024-03-21T08:41:14.261754+0000","last_peered":"2024-03-21T08:41:14.261754+0000","last_clean":"2024-03-21T08:41:14.261754+0000","last_became_active":"2024-03-21T08:41:08.005255+0000","last_became_peered":"2024-03-21T08:41:08.005255+0000","last_unstale":"2024-03-21T08:41:14.261754+0000","last_undegraded":"2024-03-21T08:41:14.261754+0000","last_fullsized":"2024-03-21T08:41:14.261754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025568+0000","last_change":"2024-03-21T08:41:07.004853+0000","last_active":"2024-03-21T08:41:13.025568+0000","last_peered":"2024-03-21T08:41:13.025568+0000","last_clean":"2024-03-21T08:41:13.025568+0000","last_became_active":"2024-03-21T08:41:07.004447+0000","last_became_peered":"2024-03-21T08:41:07.004447+0000","last_unstale":"2024-03-21T08:41:13.025568+0000","last_undegraded":"2024-03-21T08:41:13.025568+0000","last_fullsized":"2024-03-21T08:41:13.025568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263561+0000","last_change":"2024-03-21T08:41:12.057098+0000","last_active":"2024-03-21T08:41:14.263561+0000","last_peered":"2024-03-21T08:41:14.263561+0000","last_clean":"2024-03-21T08:41:14.263561+0000","last_became_active":"2024-03-21T08:41:12.056306+0000","last_became_peered":"2024-03-21T08:41:12.056306+0000","last_unstale":"2024-03-21T08:41:14.263561+0000","last_undegraded":"2024-03-21T08:41:14.263561+0000","last_fullsized":"2024-03-21T08:41:14.263561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:00:00.666748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160776+0000","last_change":"2024-03-21T08:41:07.995187+0000","last_active":"2024-03-21T08:41:13.160776+0000","last_peered":"2024-03-21T08:41:13.160776+0000","last_clean":"2024-03-21T08:41:13.160776+0000","last_became_active":"2024-03-21T08:41:07.994882+0000","last_became_peered":"2024-03-21T08:41:07.994882+0000","last_unstale":"2024-03-21T08:41:13.160776+0000","last_undegraded":"2024-03-21T08:41:13.160776+0000","last_fullsized":"2024-03-21T08:41:13.160776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:27.177399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736421+0000","last_change":"2024-03-21T08:41:07.019906+0000","last_active":"2024-03-21T08:41:13.736421+0000","last_peered":"2024-03-21T08:41:13.736421+0000","last_clean":"2024-03-21T08:41:13.736421+0000","last_became_active":"2024-03-21T08:41:07.019585+0000","last_became_peered":"2024-03-21T08:41:07.019585+0000","last_unstale":"2024-03-21T08:41:13.736421+0000","last_undegraded":"2024-03-21T08:41:13.736421+0000","last_fullsized":"2024-03-21T08:41:13.736421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161357+0000","last_change":"2024-03-21T08:41:12.337710+0000","last_active":"2024-03-21T08:41:13.161357+0000","last_peered":"2024-03-21T08:41:13.161357+0000","last_clean":"2024-03-21T08:41:13.161357+0000","last_became_active":"2024-03-21T08:41:12.336270+0000","last_became_peered":"2024-03-21T08:41:12.336270+0000","last_unstale":"2024-03-21T08:41:13.161357+0000","last_undegraded":"2024-03-21T08:41:13.161357+0000","last_fullsized":"2024-03-21T08:41:13.161357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:11.100710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261759+0000","last_change":"2024-03-21T08:41:08.015395+0000","last_active":"2024-03-21T08:41:14.261759+0000","last_peered":"2024-03-21T08:41:14.261759+0000","last_clean":"2024-03-21T08:41:14.261759+0000","last_became_active":"2024-03-21T08:41:08.015091+0000","last_became_peered":"2024-03-21T08:41:08.015091+0000","last_unstale":"2024-03-21T08:41:14.261759+0000","last_undegraded":"2024-03-21T08:41:14.261759+0000","last_fullsized":"2024-03-21T08:41:14.261759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260786+0000","last_change":"2024-03-21T08:41:07.008427+0000","last_active":"2024-03-21T08:41:14.260786+0000","last_peered":"2024-03-21T08:41:14.260786+0000","last_clean":"2024-03-21T08:41:14.260786+0000","last_became_active":"2024-03-21T08:41:07.008115+0000","last_became_peered":"2024-03-21T08:41:07.008115+0000","last_unstale":"2024-03-21T08:41:14.260786+0000","last_undegraded":"2024-03-21T08:41:14.260786+0000","last_fullsized":"2024-03-21T08:41:14.260786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736516+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:41:13.736516+0000","last_peered":"2024-03-21T08:41:13.736516+0000","last_clean":"2024-03-21T08:41:13.736516+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:41:13.736516+0000","last_undegraded":"2024-03-21T08:41:13.736516+0000","last_fullsized":"2024-03-21T08:41:13.736516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026534+0000","last_change":"2024-03-21T08:41:12.334882+0000","last_active":"2024-03-21T08:41:13.026534+0000","last_peered":"2024-03-21T08:41:13.026534+0000","last_clean":"2024-03-21T08:41:13.026534+0000","last_became_active":"2024-03-21T08:41:12.334459+0000","last_became_peered":"2024-03-21T08:41:12.334459+0000","last_unstale":"2024-03-21T08:41:13.026534+0000","last_undegraded":"2024-03-21T08:41:13.026534+0000","last_fullsized":"2024-03-21T08:41:13.026534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:17.097499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736898+0000","last_change":"2024-03-21T08:41:08.030635+0000","last_active":"2024-03-21T08:41:13.736898+0000","last_peered":"2024-03-21T08:41:13.736898+0000","last_clean":"2024-03-21T08:41:13.736898+0000","last_became_active":"2024-03-21T08:41:08.028635+0000","last_became_peered":"2024-03-21T08:41:08.028635+0000","last_unstale":"2024-03-21T08:41:13.736898+0000","last_undegraded":"2024-03-21T08:41:13.736898+0000","last_fullsized":"2024-03-21T08:41:13.736898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:14.013088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026041+0000","last_change":"2024-03-21T08:41:07.004839+0000","last_active":"2024-03-21T08:41:13.026041+0000","last_peered":"2024-03-21T08:41:13.026041+0000","last_clean":"2024-03-21T08:41:13.026041+0000","last_became_active":"2024-03-21T08:41:07.004409+0000","last_became_peered":"2024-03-21T08:41:07.004409+0000","last_unstale":"2024-03-21T08:41:13.026041+0000","last_undegraded":"2024-03-21T08:41:13.026041+0000","last_fullsized":"2024-03-21T08:41:13.026041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025597+0000","last_change":"2024-03-21T08:41:12.334998+0000","last_active":"2024-03-21T08:41:13.025597+0000","last_peered":"2024-03-21T08:41:13.025597+0000","last_clean":"2024-03-21T08:41:13.025597+0000","last_became_active":"2024-03-21T08:41:12.334610+0000","last_became_peered":"2024-03-21T08:41:12.334610+0000","last_unstale":"2024-03-21T08:41:13.025597+0000","last_undegraded":"2024-03-21T08:41:13.025597+0000","last_fullsized":"2024-03-21T08:41:13.025597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:04.540298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261002+0000","last_change":"2024-03-21T08:41:08.014545+0000","last_active":"2024-03-21T08:41:14.261002+0000","last_peered":"2024-03-21T08:41:14.261002+0000","last_clean":"2024-03-21T08:41:14.261002+0000","last_became_active":"2024-03-21T08:41:08.013815+0000","last_became_peered":"2024-03-21T08:41:08.013815+0000","last_unstale":"2024-03-21T08:41:14.261002+0000","last_undegraded":"2024-03-21T08:41:14.261002+0000","last_fullsized":"2024-03-21T08:41:14.261002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:07.068892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263256+0000","last_change":"2024-03-21T08:41:07.011021+0000","last_active":"2024-03-21T08:41:14.263256+0000","last_peered":"2024-03-21T08:41:14.263256+0000","last_clean":"2024-03-21T08:41:14.263256+0000","last_became_active":"2024-03-21T08:41:07.010411+0000","last_became_peered":"2024-03-21T08:41:07.010411+0000","last_unstale":"2024-03-21T08:41:14.263256+0000","last_undegraded":"2024-03-21T08:41:14.263256+0000","last_fullsized":"2024-03-21T08:41:14.263256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161403+0000","last_change":"2024-03-21T08:41:12.055570+0000","last_active":"2024-03-21T08:41:13.161403+0000","last_peered":"2024-03-21T08:41:13.161403+0000","last_clean":"2024-03-21T08:41:13.161403+0000","last_became_active":"2024-03-21T08:41:12.055174+0000","last_became_peered":"2024-03-21T08:41:12.055174+0000","last_unstale":"2024-03-21T08:41:13.161403+0000","last_undegraded":"2024-03-21T08:41:13.161403+0000","last_fullsized":"2024-03-21T08:41:13.161403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:45:15.642543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737253+0000","last_change":"2024-03-21T08:41:08.037018+0000","last_active":"2024-03-21T08:41:13.737253+0000","last_peered":"2024-03-21T08:41:13.737253+0000","last_clean":"2024-03-21T08:41:13.737253+0000","last_became_active":"2024-03-21T08:41:08.036676+0000","last_became_peered":"2024-03-21T08:41:08.036676+0000","last_unstale":"2024-03-21T08:41:13.737253+0000","last_undegraded":"2024-03-21T08:41:13.737253+0000","last_fullsized":"2024-03-21T08:41:13.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:59:56.634930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025975+0000","last_change":"2024-03-21T08:41:07.004711+0000","last_active":"2024-03-21T08:41:13.025975+0000","last_peered":"2024-03-21T08:41:13.025975+0000","last_clean":"2024-03-21T08:41:13.025975+0000","last_became_active":"2024-03-21T08:41:07.004192+0000","last_became_peered":"2024-03-21T08:41:07.004192+0000","last_unstale":"2024-03-21T08:41:13.025975+0000","last_undegraded":"2024-03-21T08:41:13.025975+0000","last_fullsized":"2024-03-21T08:41:13.025975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260477+0000","last_change":"2024-03-21T08:41:12.341167+0000","last_active":"2024-03-21T08:41:14.260477+0000","last_peered":"2024-03-21T08:41:14.260477+0000","last_clean":"2024-03-21T08:41:14.260477+0000","last_became_active":"2024-03-21T08:41:12.339602+0000","last_became_peered":"2024-03-21T08:41:12.339602+0000","last_unstale":"2024-03-21T08:41:14.260477+0000","last_undegraded":"2024-03-21T08:41:14.260477+0000","last_fullsized":"2024-03-21T08:41:14.260477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737088+0000","last_change":"2024-03-21T08:41:08.033751+0000","last_active":"2024-03-21T08:41:13.737088+0000","last_peered":"2024-03-21T08:41:13.737088+0000","last_clean":"2024-03-21T08:41:13.737088+0000","last_became_active":"2024-03-21T08:41:08.033458+0000","last_became_peered":"2024-03-21T08:41:08.033458+0000","last_unstale":"2024-03-21T08:41:13.737088+0000","last_undegraded":"2024-03-21T08:41:13.737088+0000","last_fullsized":"2024-03-21T08:41:13.737088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:13.065681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027669+0000","last_change":"2024-03-21T08:41:07.017025+0000","last_active":"2024-03-21T08:41:13.027669+0000","last_peered":"2024-03-21T08:41:13.027669+0000","last_clean":"2024-03-21T08:41:13.027669+0000","last_became_active":"2024-03-21T08:41:07.016654+0000","last_became_peered":"2024-03-21T08:41:07.016654+0000","last_unstale":"2024-03-21T08:41:13.027669+0000","last_undegraded":"2024-03-21T08:41:13.027669+0000","last_fullsized":"2024-03-21T08:41:13.027669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:10:03.436446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025227+0000","last_change":"2024-03-21T08:41:12.048497+0000","last_active":"2024-03-21T08:41:13.025227+0000","last_peered":"2024-03-21T08:41:13.025227+0000","last_clean":"2024-03-21T08:41:13.025227+0000","last_became_active":"2024-03-21T08:41:12.047987+0000","last_became_peered":"2024-03-21T08:41:12.047987+0000","last_unstale":"2024-03-21T08:41:13.025227+0000","last_undegraded":"2024-03-21T08:41:13.025227+0000","last_fullsized":"2024-03-21T08:41:13.025227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:39.659897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736537+0000","last_change":"2024-03-21T08:41:08.033999+0000","last_active":"2024-03-21T08:41:13.736537+0000","last_peered":"2024-03-21T08:41:13.736537+0000","last_clean":"2024-03-21T08:41:13.736537+0000","last_became_active":"2024-03-21T08:41:08.033015+0000","last_became_peered":"2024-03-21T08:41:08.033015+0000","last_unstale":"2024-03-21T08:41:13.736537+0000","last_undegraded":"2024-03-21T08:41:13.736537+0000","last_fullsized":"2024-03-21T08:41:13.736537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:16:25.591202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025409+0000","last_change":"2024-03-21T08:41:07.001332+0000","last_active":"2024-03-21T08:41:13.025409+0000","last_peered":"2024-03-21T08:41:13.025409+0000","last_clean":"2024-03-21T08:41:13.025409+0000","last_became_active":"2024-03-21T08:41:06.999605+0000","last_became_peered":"2024-03-21T08:41:06.999605+0000","last_unstale":"2024-03-21T08:41:13.025409+0000","last_undegraded":"2024-03-21T08:41:13.025409+0000","last_fullsized":"2024-03-21T08:41:13.025409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:06:31.096538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161359+0000","last_change":"2024-03-21T08:41:12.059105+0000","last_active":"2024-03-21T08:41:13.161359+0000","last_peered":"2024-03-21T08:41:13.161359+0000","last_clean":"2024-03-21T08:41:13.161359+0000","last_became_active":"2024-03-21T08:41:12.058643+0000","last_became_peered":"2024-03-21T08:41:12.058643+0000","last_unstale":"2024-03-21T08:41:13.161359+0000","last_undegraded":"2024-03-21T08:41:13.161359+0000","last_fullsized":"2024-03-21T08:41:13.161359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.221689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025534+0000","last_change":"2024-03-21T08:41:08.010609+0000","last_active":"2024-03-21T08:41:13.025534+0000","last_peered":"2024-03-21T08:41:13.025534+0000","last_clean":"2024-03-21T08:41:13.025534+0000","last_became_active":"2024-03-21T08:41:08.010109+0000","last_became_peered":"2024-03-21T08:41:08.010109+0000","last_unstale":"2024-03-21T08:41:13.025534+0000","last_undegraded":"2024-03-21T08:41:13.025534+0000","last_fullsized":"2024-03-21T08:41:13.025534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026017+0000","last_change":"2024-03-21T08:41:07.010384+0000","last_active":"2024-03-21T08:41:13.026017+0000","last_peered":"2024-03-21T08:41:13.026017+0000","last_clean":"2024-03-21T08:41:13.026017+0000","last_became_active":"2024-03-21T08:41:07.010019+0000","last_became_peered":"2024-03-21T08:41:07.010019+0000","last_unstale":"2024-03-21T08:41:13.026017+0000","last_undegraded":"2024-03-21T08:41:13.026017+0000","last_fullsized":"2024-03-21T08:41:13.026017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262930+0000","last_change":"2024-03-21T08:41:12.341603+0000","last_active":"2024-03-21T08:41:14.262930+0000","last_peered":"2024-03-21T08:41:14.262930+0000","last_clean":"2024-03-21T08:41:14.262930+0000","last_became_active":"2024-03-21T08:41:12.341158+0000","last_became_peered":"2024-03-21T08:41:12.341158+0000","last_unstale":"2024-03-21T08:41:14.262930+0000","last_undegraded":"2024-03-21T08:41:14.262930+0000","last_fullsized":"2024-03-21T08:41:14.262930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:02.246810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736756+0000","last_change":"2024-03-21T08:41:08.032505+0000","last_active":"2024-03-21T08:41:13.736756+0000","last_peered":"2024-03-21T08:41:13.736756+0000","last_clean":"2024-03-21T08:41:13.736756+0000","last_became_active":"2024-03-21T08:41:08.032026+0000","last_became_peered":"2024-03-21T08:41:08.032026+0000","last_unstale":"2024-03-21T08:41:13.736756+0000","last_undegraded":"2024-03-21T08:41:13.736756+0000","last_fullsized":"2024-03-21T08:41:13.736756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:21.306665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260844+0000","last_change":"2024-03-21T08:41:07.003264+0000","last_active":"2024-03-21T08:41:14.260844+0000","last_peered":"2024-03-21T08:41:14.260844+0000","last_clean":"2024-03-21T08:41:14.260844+0000","last_became_active":"2024-03-21T08:41:07.002969+0000","last_became_peered":"2024-03-21T08:41:07.002969+0000","last_unstale":"2024-03-21T08:41:14.260844+0000","last_undegraded":"2024-03-21T08:41:14.260844+0000","last_fullsized":"2024-03-21T08:41:14.260844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:19.390331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262423+0000","last_change":"2024-03-21T08:41:12.336949+0000","last_active":"2024-03-21T08:41:14.262423+0000","last_peered":"2024-03-21T08:41:14.262423+0000","last_clean":"2024-03-21T08:41:14.262423+0000","last_became_active":"2024-03-21T08:41:12.335654+0000","last_became_peered":"2024-03-21T08:41:12.335654+0000","last_unstale":"2024-03-21T08:41:14.262423+0000","last_undegraded":"2024-03-21T08:41:14.262423+0000","last_fullsized":"2024-03-21T08:41:14.262423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:54:40.677358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160969+0000","last_change":"2024-03-21T08:41:08.020852+0000","last_active":"2024-03-21T08:41:13.160969+0000","last_peered":"2024-03-21T08:41:13.160969+0000","last_clean":"2024-03-21T08:41:13.160969+0000","last_became_active":"2024-03-21T08:41:08.020547+0000","last_became_peered":"2024-03-21T08:41:08.020547+0000","last_unstale":"2024-03-21T08:41:13.160969+0000","last_undegraded":"2024-03-21T08:41:13.160969+0000","last_fullsized":"2024-03-21T08:41:13.160969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:56.518303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161459+0000","last_change":"2024-03-21T08:41:07.013979+0000","last_active":"2024-03-21T08:41:13.161459+0000","last_peered":"2024-03-21T08:41:13.161459+0000","last_clean":"2024-03-21T08:41:13.161459+0000","last_became_active":"2024-03-21T08:41:07.013725+0000","last_became_peered":"2024-03-21T08:41:07.013725+0000","last_unstale":"2024-03-21T08:41:13.161459+0000","last_undegraded":"2024-03-21T08:41:13.161459+0000","last_fullsized":"2024-03-21T08:41:13.161459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:03:13.978443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261111+0000","last_change":"2024-03-21T08:41:12.335008+0000","last_active":"2024-03-21T08:41:14.261111+0000","last_peered":"2024-03-21T08:41:14.261111+0000","last_clean":"2024-03-21T08:41:14.261111+0000","last_became_active":"2024-03-21T08:41:12.334635+0000","last_became_peered":"2024-03-21T08:41:12.334635+0000","last_unstale":"2024-03-21T08:41:14.261111+0000","last_undegraded":"2024-03-21T08:41:14.261111+0000","last_fullsized":"2024-03-21T08:41:14.261111+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736888+0000","last_change":"2024-03-21T08:41:08.026214+0000","last_active":"2024-03-21T08:41:13.736888+0000","last_peered":"2024-03-21T08:41:13.736888+0000","last_clean":"2024-03-21T08:41:13.736888+0000","last_became_active":"2024-03-21T08:41:08.025686+0000","last_became_peered":"2024-03-21T08:41:08.025686+0000","last_unstale":"2024-03-21T08:41:13.736888+0000","last_undegraded":"2024-03-21T08:41:13.736888+0000","last_fullsized":"2024-03-21T08:41:13.736888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:32:26.184390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:23.028503+0000","last_change":"2024-03-21T08:41:07.005099+0000","last_active":"2024-03-21T08:41:23.028503+0000","last_peered":"2024-03-21T08:41:23.028503+0000","last_clean":"2024-03-21T08:41:23.028503+0000","last_became_active":"2024-03-21T08:41:07.004825+0000","last_became_peered":"2024-03-21T08:41:07.004825+0000","last_unstale":"2024-03-21T08:41:23.028503+0000","last_undegraded":"2024-03-21T08:41:23.028503+0000","last_fullsized":"2024-03-21T08:41:23.028503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262224+0000","last_change":"2024-03-21T08:41:12.057213+0000","last_active":"2024-03-21T08:41:14.262224+0000","last_peered":"2024-03-21T08:41:14.262224+0000","last_clean":"2024-03-21T08:41:14.262224+0000","last_became_active":"2024-03-21T08:41:12.056539+0000","last_became_peered":"2024-03-21T08:41:12.056539+0000","last_unstale":"2024-03-21T08:41:14.262224+0000","last_undegraded":"2024-03-21T08:41:14.262224+0000","last_fullsized":"2024-03-21T08:41:14.262224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:35.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027129+0000","last_change":"2024-03-21T08:41:08.018576+0000","last_active":"2024-03-21T08:41:13.027129+0000","last_peered":"2024-03-21T08:41:13.027129+0000","last_clean":"2024-03-21T08:41:13.027129+0000","last_became_active":"2024-03-21T08:41:08.018277+0000","last_became_peered":"2024-03-21T08:41:08.018277+0000","last_unstale":"2024-03-21T08:41:13.027129+0000","last_undegraded":"2024-03-21T08:41:13.027129+0000","last_fullsized":"2024-03-21T08:41:13.027129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:00:26.730038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263165+0000","last_change":"2024-03-21T08:41:07.013500+0000","last_active":"2024-03-21T08:41:14.263165+0000","last_peered":"2024-03-21T08:41:14.263165+0000","last_clean":"2024-03-21T08:41:14.263165+0000","last_became_active":"2024-03-21T08:41:07.010794+0000","last_became_peered":"2024-03-21T08:41:07.010794+0000","last_unstale":"2024-03-21T08:41:14.263165+0000","last_undegraded":"2024-03-21T08:41:14.263165+0000","last_fullsized":"2024-03-21T08:41:14.263165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027024+0000","last_change":"2024-03-21T08:41:12.049156+0000","last_active":"2024-03-21T08:41:13.027024+0000","last_peered":"2024-03-21T08:41:13.027024+0000","last_clean":"2024-03-21T08:41:13.027024+0000","last_became_active":"2024-03-21T08:41:12.048737+0000","last_became_peered":"2024-03-21T08:41:12.048737+0000","last_unstale":"2024-03-21T08:41:13.027024+0000","last_undegraded":"2024-03-21T08:41:13.027024+0000","last_fullsized":"2024-03-21T08:41:13.027024+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:12.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161355+0000","last_change":"2024-03-21T08:41:08.012883+0000","last_active":"2024-03-21T08:41:13.161355+0000","last_peered":"2024-03-21T08:41:13.161355+0000","last_clean":"2024-03-21T08:41:13.161355+0000","last_became_active":"2024-03-21T08:41:08.012565+0000","last_became_peered":"2024-03-21T08:41:08.012565+0000","last_unstale":"2024-03-21T08:41:13.161355+0000","last_undegraded":"2024-03-21T08:41:13.161355+0000","last_fullsized":"2024-03-21T08:41:13.161355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261096+0000","last_change":"2024-03-21T08:41:07.013115+0000","last_active":"2024-03-21T08:41:14.261096+0000","last_peered":"2024-03-21T08:41:14.261096+0000","last_clean":"2024-03-21T08:41:14.261096+0000","last_became_active":"2024-03-21T08:41:07.012884+0000","last_became_peered":"2024-03-21T08:41:07.012884+0000","last_unstale":"2024-03-21T08:41:14.261096+0000","last_undegraded":"2024-03-21T08:41:14.261096+0000","last_fullsized":"2024-03-21T08:41:14.261096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:39.115443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261540+0000","last_change":"2024-03-21T08:41:12.051266+0000","last_active":"2024-03-21T08:41:14.261540+0000","last_peered":"2024-03-21T08:41:14.261540+0000","last_clean":"2024-03-21T08:41:14.261540+0000","last_became_active":"2024-03-21T08:41:12.050452+0000","last_became_peered":"2024-03-21T08:41:12.050452+0000","last_unstale":"2024-03-21T08:41:14.261540+0000","last_undegraded":"2024-03-21T08:41:14.261540+0000","last_fullsized":"2024-03-21T08:41:14.261540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261053+0000","last_change":"2024-03-21T08:41:08.029184+0000","last_active":"2024-03-21T08:41:14.261053+0000","last_peered":"2024-03-21T08:41:14.261053+0000","last_clean":"2024-03-21T08:41:14.261053+0000","last_became_active":"2024-03-21T08:41:08.028897+0000","last_became_peered":"2024-03-21T08:41:08.028897+0000","last_unstale":"2024-03-21T08:41:14.261053+0000","last_undegraded":"2024-03-21T08:41:14.261053+0000","last_fullsized":"2024-03-21T08:41:14.261053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:59.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026681+0000","last_change":"2024-03-21T08:41:07.021159+0000","last_active":"2024-03-21T08:41:13.026681+0000","last_peered":"2024-03-21T08:41:13.026681+0000","last_clean":"2024-03-21T08:41:13.026681+0000","last_became_active":"2024-03-21T08:41:07.020879+0000","last_became_peered":"2024-03-21T08:41:07.020879+0000","last_unstale":"2024-03-21T08:41:13.026681+0000","last_undegraded":"2024-03-21T08:41:13.026681+0000","last_fullsized":"2024-03-21T08:41:13.026681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261676+0000","last_change":"2024-03-21T08:41:12.340577+0000","last_active":"2024-03-21T08:41:14.261676+0000","last_peered":"2024-03-21T08:41:14.261676+0000","last_clean":"2024-03-21T08:41:14.261676+0000","last_became_active":"2024-03-21T08:41:12.339629+0000","last_became_peered":"2024-03-21T08:41:12.339629+0000","last_unstale":"2024-03-21T08:41:14.261676+0000","last_undegraded":"2024-03-21T08:41:14.261676+0000","last_fullsized":"2024-03-21T08:41:14.261676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:42.494428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161192+0000","last_change":"2024-03-21T08:41:12.340677+0000","last_active":"2024-03-21T08:41:13.161192+0000","last_peered":"2024-03-21T08:41:13.161192+0000","last_clean":"2024-03-21T08:41:13.161192+0000","last_became_active":"2024-03-21T08:41:12.340153+0000","last_became_peered":"2024-03-21T08:41:12.340153+0000","last_unstale":"2024-03-21T08:41:13.161192+0000","last_undegraded":"2024-03-21T08:41:13.161192+0000","last_fullsized":"2024-03-21T08:41:13.161192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:48.677726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261917+0000","last_change":"2024-03-21T08:41:08.004430+0000","last_active":"2024-03-21T08:41:14.261917+0000","last_peered":"2024-03-21T08:41:14.261917+0000","last_clean":"2024-03-21T08:41:14.261917+0000","last_became_active":"2024-03-21T08:41:08.003935+0000","last_became_peered":"2024-03-21T08:41:08.003935+0000","last_unstale":"2024-03-21T08:41:14.261917+0000","last_undegraded":"2024-03-21T08:41:14.261917+0000","last_fullsized":"2024-03-21T08:41:14.261917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260592+0000","last_change":"2024-03-21T08:41:07.011532+0000","last_active":"2024-03-21T08:41:14.260592+0000","last_peered":"2024-03-21T08:41:14.260592+0000","last_clean":"2024-03-21T08:41:14.260592+0000","last_became_active":"2024-03-21T08:41:07.008460+0000","last_became_peered":"2024-03-21T08:41:07.008460+0000","last_unstale":"2024-03-21T08:41:14.260592+0000","last_undegraded":"2024-03-21T08:41:14.260592+0000","last_fullsized":"2024-03-21T08:41:14.260592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026625+0000","last_change":"2024-03-21T08:41:12.054672+0000","last_active":"2024-03-21T08:41:13.026625+0000","last_peered":"2024-03-21T08:41:13.026625+0000","last_clean":"2024-03-21T08:41:13.026625+0000","last_became_active":"2024-03-21T08:41:12.054257+0000","last_became_peered":"2024-03-21T08:41:12.054257+0000","last_unstale":"2024-03-21T08:41:13.026625+0000","last_undegraded":"2024-03-21T08:41:13.026625+0000","last_fullsized":"2024-03-21T08:41:13.026625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261534+0000","last_change":"2024-03-21T08:41:08.015362+0000","last_active":"2024-03-21T08:41:14.261534+0000","last_peered":"2024-03-21T08:41:14.261534+0000","last_clean":"2024-03-21T08:41:14.261534+0000","last_became_active":"2024-03-21T08:41:08.015110+0000","last_became_peered":"2024-03-21T08:41:08.015110+0000","last_unstale":"2024-03-21T08:41:14.261534+0000","last_undegraded":"2024-03-21T08:41:14.261534+0000","last_fullsized":"2024-03-21T08:41:14.261534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261823+0000","last_change":"2024-03-21T08:41:07.001659+0000","last_active":"2024-03-21T08:41:14.261823+0000","last_peered":"2024-03-21T08:41:14.261823+0000","last_clean":"2024-03-21T08:41:14.261823+0000","last_became_active":"2024-03-21T08:41:07.001354+0000","last_became_peered":"2024-03-21T08:41:07.001354+0000","last_unstale":"2024-03-21T08:41:14.261823+0000","last_undegraded":"2024-03-21T08:41:14.261823+0000","last_fullsized":"2024-03-21T08:41:14.261823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.256597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026173+0000","last_change":"2024-03-21T08:41:07.009459+0000","last_active":"2024-03-21T08:41:13.026173+0000","last_peered":"2024-03-21T08:41:13.026173+0000","last_clean":"2024-03-21T08:41:13.026173+0000","last_became_active":"2024-03-21T08:41:07.009113+0000","last_became_peered":"2024-03-21T08:41:07.009113+0000","last_unstale":"2024-03-21T08:41:13.026173+0000","last_undegraded":"2024-03-21T08:41:13.026173+0000","last_fullsized":"2024-03-21T08:41:13.026173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161366+0000","last_change":"2024-03-21T08:41:07.996784+0000","last_active":"2024-03-21T08:41:13.161366+0000","last_peered":"2024-03-21T08:41:13.161366+0000","last_clean":"2024-03-21T08:41:13.161366+0000","last_became_active":"2024-03-21T08:41:07.996521+0000","last_became_peered":"2024-03-21T08:41:07.996521+0000","last_unstale":"2024-03-21T08:41:13.161366+0000","last_undegraded":"2024-03-21T08:41:13.161366+0000","last_fullsized":"2024-03-21T08:41:13.161366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025773+0000","last_change":"2024-03-21T08:41:12.052094+0000","last_active":"2024-03-21T08:41:13.025773+0000","last_peered":"2024-03-21T08:41:13.025773+0000","last_clean":"2024-03-21T08:41:13.025773+0000","last_became_active":"2024-03-21T08:41:12.051727+0000","last_became_peered":"2024-03-21T08:41:12.051727+0000","last_unstale":"2024-03-21T08:41:13.025773+0000","last_undegraded":"2024-03-21T08:41:13.025773+0000","last_fullsized":"2024-03-21T08:41:13.025773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:06:49.718295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262269+0000","last_change":"2024-03-21T08:41:07.012156+0000","last_active":"2024-03-21T08:41:14.262269+0000","last_peered":"2024-03-21T08:41:14.262269+0000","last_clean":"2024-03-21T08:41:14.262269+0000","last_became_active":"2024-03-21T08:41:07.011881+0000","last_became_peered":"2024-03-21T08:41:07.011881+0000","last_unstale":"2024-03-21T08:41:14.262269+0000","last_undegraded":"2024-03-21T08:41:14.262269+0000","last_fullsized":"2024-03-21T08:41:14.262269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027548+0000","last_change":"2024-03-21T08:41:08.002324+0000","last_active":"2024-03-21T08:41:13.027548+0000","last_peered":"2024-03-21T08:41:13.027548+0000","last_clean":"2024-03-21T08:41:13.027548+0000","last_became_active":"2024-03-21T08:41:08.002038+0000","last_became_peered":"2024-03-21T08:41:08.002038+0000","last_unstale":"2024-03-21T08:41:13.027548+0000","last_undegraded":"2024-03-21T08:41:13.027548+0000","last_fullsized":"2024-03-21T08:41:13.027548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:20.396047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262336+0000","last_change":"2024-03-21T08:41:12.058878+0000","last_active":"2024-03-21T08:41:14.262336+0000","last_peered":"2024-03-21T08:41:14.262336+0000","last_clean":"2024-03-21T08:41:14.262336+0000","last_became_active":"2024-03-21T08:41:12.058446+0000","last_became_peered":"2024-03-21T08:41:12.058446+0000","last_unstale":"2024-03-21T08:41:14.262336+0000","last_undegraded":"2024-03-21T08:41:14.262336+0000","last_fullsized":"2024-03-21T08:41:14.262336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:59:29.179384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025278+0000","last_change":"2024-03-21T08:41:08.010406+0000","last_active":"2024-03-21T08:41:13.025278+0000","last_peered":"2024-03-21T08:41:13.025278+0000","last_clean":"2024-03-21T08:41:13.025278+0000","last_became_active":"2024-03-21T08:41:08.009763+0000","last_became_peered":"2024-03-21T08:41:08.009763+0000","last_unstale":"2024-03-21T08:41:13.025278+0000","last_undegraded":"2024-03-21T08:41:13.025278+0000","last_fullsized":"2024-03-21T08:41:13.025278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025559+0000","last_change":"2024-03-21T08:41:08.009958+0000","last_active":"2024-03-21T08:41:13.025559+0000","last_peered":"2024-03-21T08:41:13.025559+0000","last_clean":"2024-03-21T08:41:13.025559+0000","last_became_active":"2024-03-21T08:41:08.009691+0000","last_became_peered":"2024-03-21T08:41:08.009691+0000","last_unstale":"2024-03-21T08:41:13.025559+0000","last_undegraded":"2024-03-21T08:41:13.025559+0000","last_fullsized":"2024-03-21T08:41:13.025559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:46.141146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260744+0000","last_change":"2024-03-21T08:41:08.004794+0000","last_active":"2024-03-21T08:41:14.260744+0000","last_peered":"2024-03-21T08:41:14.260744+0000","last_clean":"2024-03-21T08:41:14.260744+0000","last_became_active":"2024-03-21T08:41:08.003470+0000","last_became_peered":"2024-03-21T08:41:08.003470+0000","last_unstale":"2024-03-21T08:41:14.260744+0000","last_undegraded":"2024-03-21T08:41:14.260744+0000","last_fullsized":"2024-03-21T08:41:14.260744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161263+0000","last_change":"2024-03-21T08:41:08.014560+0000","last_active":"2024-03-21T08:41:13.161263+0000","last_peered":"2024-03-21T08:41:13.161263+0000","last_clean":"2024-03-21T08:41:13.161263+0000","last_became_active":"2024-03-21T08:41:08.014307+0000","last_became_peered":"2024-03-21T08:41:08.014307+0000","last_unstale":"2024-03-21T08:41:13.161263+0000","last_undegraded":"2024-03-21T08:41:13.161263+0000","last_fullsized":"2024-03-21T08:41:13.161263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262543+0000","last_change":"2024-03-21T08:41:08.026095+0000","last_active":"2024-03-21T08:41:14.262543+0000","last_peered":"2024-03-21T08:41:14.262543+0000","last_clean":"2024-03-21T08:41:14.262543+0000","last_became_active":"2024-03-21T08:41:08.025719+0000","last_became_peered":"2024-03-21T08:41:08.025719+0000","last_unstale":"2024-03-21T08:41:14.262543+0000","last_undegraded":"2024-03-21T08:41:14.262543+0000","last_fullsized":"2024-03-21T08:41:14.262543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262685+0000","last_change":"2024-03-21T08:41:08.007171+0000","last_active":"2024-03-21T08:41:14.262685+0000","last_peered":"2024-03-21T08:41:14.262685+0000","last_clean":"2024-03-21T08:41:14.262685+0000","last_became_active":"2024-03-21T08:41:08.006702+0000","last_became_peered":"2024-03-21T08:41:08.006702+0000","last_unstale":"2024-03-21T08:41:14.262685+0000","last_undegraded":"2024-03-21T08:41:14.262685+0000","last_fullsized":"2024-03-21T08:41:14.262685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026846+0000","last_change":"2024-03-21T08:41:08.029843+0000","last_active":"2024-03-21T08:41:13.026846+0000","last_peered":"2024-03-21T08:41:13.026846+0000","last_clean":"2024-03-21T08:41:13.026846+0000","last_became_active":"2024-03-21T08:41:08.029528+0000","last_became_peered":"2024-03-21T08:41:08.029528+0000","last_unstale":"2024-03-21T08:41:13.026846+0000","last_undegraded":"2024-03-21T08:41:13.026846+0000","last_fullsized":"2024-03-21T08:41:13.026846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262888+0000","last_change":"2024-03-21T08:41:08.013879+0000","last_active":"2024-03-21T08:41:14.262888+0000","last_peered":"2024-03-21T08:41:14.262888+0000","last_clean":"2024-03-21T08:41:14.262888+0000","last_became_active":"2024-03-21T08:41:08.013610+0000","last_became_peered":"2024-03-21T08:41:08.013610+0000","last_unstale":"2024-03-21T08:41:14.262888+0000","last_undegraded":"2024-03-21T08:41:14.262888+0000","last_fullsized":"2024-03-21T08:41:14.262888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260551+0000","last_change":"2024-03-21T08:41:08.001171+0000","last_active":"2024-03-21T08:41:14.260551+0000","last_peered":"2024-03-21T08:41:14.260551+0000","last_clean":"2024-03-21T08:41:14.260551+0000","last_became_active":"2024-03-21T08:41:08.000833+0000","last_became_peered":"2024-03-21T08:41:08.000833+0000","last_unstale":"2024-03-21T08:41:14.260551+0000","last_undegraded":"2024-03-21T08:41:14.260551+0000","last_fullsized":"2024-03-21T08:41:14.260551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261730+0000","last_change":"2024-03-21T08:41:08.014405+0000","last_active":"2024-03-21T08:41:14.261730+0000","last_peered":"2024-03-21T08:41:14.261730+0000","last_clean":"2024-03-21T08:41:14.261730+0000","last_became_active":"2024-03-21T08:41:08.014151+0000","last_became_peered":"2024-03-21T08:41:08.014151+0000","last_unstale":"2024-03-21T08:41:14.261730+0000","last_undegraded":"2024-03-21T08:41:14.261730+0000","last_fullsized":"2024-03-21T08:41:14.261730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260722+0000","last_change":"2024-03-21T08:41:08.019319+0000","last_active":"2024-03-21T08:41:14.260722+0000","last_peered":"2024-03-21T08:41:14.260722+0000","last_clean":"2024-03-21T08:41:14.260722+0000","last_became_active":"2024-03-21T08:41:08.019027+0000","last_became_peered":"2024-03-21T08:41:08.019027+0000","last_unstale":"2024-03-21T08:41:14.260722+0000","last_undegraded":"2024-03-21T08:41:14.260722+0000","last_fullsized":"2024-03-21T08:41:14.260722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161113+0000","last_change":"2024-03-21T08:41:08.018077+0000","last_active":"2024-03-21T08:41:13.161113+0000","last_peered":"2024-03-21T08:41:13.161113+0000","last_clean":"2024-03-21T08:41:13.161113+0000","last_became_active":"2024-03-21T08:41:08.017564+0000","last_became_peered":"2024-03-21T08:41:08.017564+0000","last_unstale":"2024-03-21T08:41:13.161113+0000","last_undegraded":"2024-03-21T08:41:13.161113+0000","last_fullsized":"2024-03-21T08:41:13.161113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026399+0000","last_change":"2024-03-21T08:41:08.018108+0000","last_active":"2024-03-21T08:41:13.026399+0000","last_peered":"2024-03-21T08:41:13.026399+0000","last_clean":"2024-03-21T08:41:13.026399+0000","last_became_active":"2024-03-21T08:41:08.017766+0000","last_became_peered":"2024-03-21T08:41:08.017766+0000","last_unstale":"2024-03-21T08:41:13.026399+0000","last_undegraded":"2024-03-21T08:41:13.026399+0000","last_fullsized":"2024-03-21T08:41:13.026399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737069+0000","last_change":"2024-03-21T08:41:08.021075+0000","last_active":"2024-03-21T08:41:13.737069+0000","last_peered":"2024-03-21T08:41:13.737069+0000","last_clean":"2024-03-21T08:41:13.737069+0000","last_became_active":"2024-03-21T08:41:08.020798+0000","last_became_peered":"2024-03-21T08:41:08.020798+0000","last_unstale":"2024-03-21T08:41:13.737069+0000","last_undegraded":"2024-03-21T08:41:13.737069+0000","last_fullsized":"2024-03-21T08:41:13.737069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:06.452855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025886+0000","last_change":"2024-03-21T08:41:08.008591+0000","last_active":"2024-03-21T08:41:13.025886+0000","last_peered":"2024-03-21T08:41:13.025886+0000","last_clean":"2024-03-21T08:41:13.025886+0000","last_became_active":"2024-03-21T08:41:08.008328+0000","last_became_peered":"2024-03-21T08:41:08.008328+0000","last_unstale":"2024-03-21T08:41:13.025886+0000","last_undegraded":"2024-03-21T08:41:13.025886+0000","last_fullsized":"2024-03-21T08:41:13.025886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:23:00.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261256+0000","last_change":"2024-03-21T08:41:08.023131+0000","last_active":"2024-03-21T08:41:14.261256+0000","last_peered":"2024-03-21T08:41:14.261256+0000","last_clean":"2024-03-21T08:41:14.261256+0000","last_became_active":"2024-03-21T08:41:08.022844+0000","last_became_peered":"2024-03-21T08:41:08.022844+0000","last_unstale":"2024-03-21T08:41:14.261256+0000","last_undegraded":"2024-03-21T08:41:14.261256+0000","last_fullsized":"2024-03-21T08:41:14.261256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:44.645908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260479+0000","last_change":"2024-03-21T08:41:08.008995+0000","last_active":"2024-03-21T08:41:14.260479+0000","last_peered":"2024-03-21T08:41:14.260479+0000","last_clean":"2024-03-21T08:41:14.260479+0000","last_became_active":"2024-03-21T08:41:08.008707+0000","last_became_peered":"2024-03-21T08:41:08.008707+0000","last_unstale":"2024-03-21T08:41:14.260479+0000","last_undegraded":"2024-03-21T08:41:14.260479+0000","last_fullsized":"2024-03-21T08:41:14.260479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261974+0000","last_change":"2024-03-21T08:41:08.007429+0000","last_active":"2024-03-21T08:41:14.261974+0000","last_peered":"2024-03-21T08:41:14.261974+0000","last_clean":"2024-03-21T08:41:14.261974+0000","last_became_active":"2024-03-21T08:41:08.007118+0000","last_became_peered":"2024-03-21T08:41:08.007118+0000","last_unstale":"2024-03-21T08:41:14.261974+0000","last_undegraded":"2024-03-21T08:41:14.261974+0000","last_fullsized":"2024-03-21T08:41:14.261974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160940+0000","last_change":"2024-03-21T08:41:08.002687+0000","last_active":"2024-03-21T08:41:13.160940+0000","last_peered":"2024-03-21T08:41:13.160940+0000","last_clean":"2024-03-21T08:41:13.160940+0000","last_became_active":"2024-03-21T08:41:08.002397+0000","last_became_peered":"2024-03-21T08:41:08.002397+0000","last_unstale":"2024-03-21T08:41:13.160940+0000","last_undegraded":"2024-03-21T08:41:13.160940+0000","last_fullsized":"2024-03-21T08:41:13.160940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:12:01.877634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262084+0000","last_change":"2024-03-21T08:41:08.011489+0000","last_active":"2024-03-21T08:41:14.262084+0000","last_peered":"2024-03-21T08:41:14.262084+0000","last_clean":"2024-03-21T08:41:14.262084+0000","last_became_active":"2024-03-21T08:41:08.011227+0000","last_became_peered":"2024-03-21T08:41:08.011227+0000","last_unstale":"2024-03-21T08:41:14.262084+0000","last_undegraded":"2024-03-21T08:41:14.262084+0000","last_fullsized":"2024-03-21T08:41:14.262084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027108+0000","last_change":"2024-03-21T08:41:08.010854+0000","last_active":"2024-03-21T08:41:13.027108+0000","last_peered":"2024-03-21T08:41:13.027108+0000","last_clean":"2024-03-21T08:41:13.027108+0000","last_became_active":"2024-03-21T08:41:08.010557+0000","last_became_peered":"2024-03-21T08:41:08.010557+0000","last_unstale":"2024-03-21T08:41:13.027108+0000","last_undegraded":"2024-03-21T08:41:13.027108+0000","last_fullsized":"2024-03-21T08:41:13.027108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:01.730277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026506+0000","last_change":"2024-03-21T08:41:08.014812+0000","last_active":"2024-03-21T08:41:13.026506+0000","last_peered":"2024-03-21T08:41:13.026506+0000","last_clean":"2024-03-21T08:41:13.026506+0000","last_became_active":"2024-03-21T08:41:08.014508+0000","last_became_peered":"2024-03-21T08:41:08.014508+0000","last_unstale":"2024-03-21T08:41:13.026506+0000","last_undegraded":"2024-03-21T08:41:13.026506+0000","last_fullsized":"2024-03-21T08:41:13.026506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026766+0000","last_change":"2024-03-21T08:41:08.013816+0000","last_active":"2024-03-21T08:41:13.026766+0000","last_peered":"2024-03-21T08:41:13.026766+0000","last_clean":"2024-03-21T08:41:13.026766+0000","last_became_active":"2024-03-21T08:41:08.013585+0000","last_became_peered":"2024-03-21T08:41:08.013585+0000","last_unstale":"2024-03-21T08:41:13.026766+0000","last_undegraded":"2024-03-21T08:41:13.026766+0000","last_fullsized":"2024-03-21T08:41:13.026766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737247+0000","last_change":"2024-03-21T08:41:08.031989+0000","last_active":"2024-03-21T08:41:13.737247+0000","last_peered":"2024-03-21T08:41:13.737247+0000","last_clean":"2024-03-21T08:41:13.737247+0000","last_became_active":"2024-03-21T08:41:08.031755+0000","last_became_peered":"2024-03-21T08:41:08.031755+0000","last_unstale":"2024-03-21T08:41:13.737247+0000","last_undegraded":"2024-03-21T08:41:13.737247+0000","last_fullsized":"2024-03-21T08:41:13.737247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:37.172205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261442+0000","last_change":"2024-03-21T08:41:08.015070+0000","last_active":"2024-03-21T08:41:14.261442+0000","last_peered":"2024-03-21T08:41:14.261442+0000","last_clean":"2024-03-21T08:41:14.261442+0000","last_became_active":"2024-03-21T08:41:08.012083+0000","last_became_peered":"2024-03-21T08:41:08.012083+0000","last_unstale":"2024-03-21T08:41:14.261442+0000","last_undegraded":"2024-03-21T08:41:14.261442+0000","last_fullsized":"2024-03-21T08:41:14.261442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736535+0000","last_change":"2024-03-21T08:41:08.020740+0000","last_active":"2024-03-21T08:41:13.736535+0000","last_peered":"2024-03-21T08:41:13.736535+0000","last_clean":"2024-03-21T08:41:13.736535+0000","last_became_active":"2024-03-21T08:41:08.020396+0000","last_became_peered":"2024-03-21T08:41:08.020396+0000","last_unstale":"2024-03-21T08:41:13.736535+0000","last_undegraded":"2024-03-21T08:41:13.736535+0000","last_fullsized":"2024-03-21T08:41:13.736535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:59:37.815273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260654+0000","last_change":"2024-03-21T08:41:08.010314+0000","last_active":"2024-03-21T08:41:14.260654+0000","last_peered":"2024-03-21T08:41:14.260654+0000","last_clean":"2024-03-21T08:41:14.260654+0000","last_became_active":"2024-03-21T08:41:08.009693+0000","last_became_peered":"2024-03-21T08:41:08.009693+0000","last_unstale":"2024-03-21T08:41:14.260654+0000","last_undegraded":"2024-03-21T08:41:14.260654+0000","last_fullsized":"2024-03-21T08:41:14.260654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736584+0000","last_change":"2024-03-21T08:41:08.020466+0000","last_active":"2024-03-21T08:41:13.736584+0000","last_peered":"2024-03-21T08:41:13.736584+0000","last_clean":"2024-03-21T08:41:13.736584+0000","last_became_active":"2024-03-21T08:41:08.020169+0000","last_became_peered":"2024-03-21T08:41:08.020169+0000","last_unstale":"2024-03-21T08:41:13.736584+0000","last_undegraded":"2024-03-21T08:41:13.736584+0000","last_fullsized":"2024-03-21T08:41:13.736584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:00.508480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260900+0000","last_change":"2024-03-21T08:41:08.014151+0000","last_active":"2024-03-21T08:41:14.260900+0000","last_peered":"2024-03-21T08:41:14.260900+0000","last_clean":"2024-03-21T08:41:14.260900+0000","last_became_active":"2024-03-21T08:41:08.013319+0000","last_became_peered":"2024-03-21T08:41:08.013319+0000","last_unstale":"2024-03-21T08:41:14.260900+0000","last_undegraded":"2024-03-21T08:41:14.260900+0000","last_fullsized":"2024-03-21T08:41:14.260900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:16.725366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161103+0000","last_change":"2024-03-21T08:41:08.002931+0000","last_active":"2024-03-21T08:41:13.161103+0000","last_peered":"2024-03-21T08:41:13.161103+0000","last_clean":"2024-03-21T08:41:13.161103+0000","last_became_active":"2024-03-21T08:41:08.001953+0000","last_became_peered":"2024-03-21T08:41:08.001953+0000","last_unstale":"2024-03-21T08:41:13.161103+0000","last_undegraded":"2024-03-21T08:41:13.161103+0000","last_fullsized":"2024-03-21T08:41:13.161103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263385+0000","last_change":"2024-03-21T08:41:08.005883+0000","last_active":"2024-03-21T08:41:14.263385+0000","last_peered":"2024-03-21T08:41:14.263385+0000","last_clean":"2024-03-21T08:41:14.263385+0000","last_became_active":"2024-03-21T08:41:08.005503+0000","last_became_peered":"2024-03-21T08:41:08.005503+0000","last_unstale":"2024-03-21T08:41:14.263385+0000","last_undegraded":"2024-03-21T08:41:14.263385+0000","last_fullsized":"2024-03-21T08:41:14.263385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737138+0000","last_change":"2024-03-21T08:41:08.030286+0000","last_active":"2024-03-21T08:41:13.737138+0000","last_peered":"2024-03-21T08:41:13.737138+0000","last_clean":"2024-03-21T08:41:13.737138+0000","last_became_active":"2024-03-21T08:41:08.029830+0000","last_became_peered":"2024-03-21T08:41:08.029830+0000","last_unstale":"2024-03-21T08:41:13.737138+0000","last_undegraded":"2024-03-21T08:41:13.737138+0000","last_fullsized":"2024-03-21T08:41:13.737138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:31.510333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":693167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":102377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":693223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:41:27.595 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-21T08:41:27.595 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T08:41:27.595 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-21T08:41:27.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-21T08:41:27.596 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-21T08:41:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.766+0000 7f6d94a00640 1 Processor -- start 2024-03-21T08:41:27.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 -- start start 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d9015f260 0x7f6d900cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d9009a290 0x7f6d900c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900cdb50 con 0x7f6d900cd5d0 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d900c5530 con 0x7f6d9009a290 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d94a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900cde30 con 0x7f6d9015f260 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d9009a290 0x7f6d900c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d9009a290 0x7f6d900c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51398/0 (socket says 172.21.15.28:51398) 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 -- 172.21.15.28:0/3964733103 learned_addr learned my addr 172.21.15.28:0/3964733103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:27.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8de00640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d9015f260 0x7f6d900cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 -- 172.21.15.28:0/3964733103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d9015f260 msgr2=0x7f6d900cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:27.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d9015f260 0x7f6d900cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 -- 172.21.15.28:0/3964733103 >> v1:172.21.15.151:6789/0 conn(0x7f6d900cd5d0 legacy=0x7f6d90099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8e800640 1 -- 172.21.15.28:0/3964733103 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900c5780 con 0x7f6d9009a290 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.770+0000 7f6d8d400640 1 -- 172.21.15.28:0/3964733103 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690295574 0 0) 0x7f6d900cdb50 con 0x7f6d900cd5d0 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d8e800640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9009a290 0x7f6d900c5f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e50 tx=0x7f6d8801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7de6405d37b7a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d8d400640 1 -- 172.21.15.28:0/3964733103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880a5850 con 0x7f6d9009a290 2024-03-21T08:41:27.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d8d400640 1 -- 172.21.15.28:0/3964733103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d880a59f0 con 0x7f6d9009a290 2024-03-21T08:41:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d8d400640 1 -- 172.21.15.28:0/3964733103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880a5dc0 con 0x7f6d9009a290 2024-03-21T08:41:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- 172.21.15.28:0/3964733103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9009a290 msgr2=0x7f6d900c5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9009a290 0x7f6d900c5f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009e50 tx=0x7f6d8801c970 comp rx=0 tx=0).stop 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- 172.21.15.28:0/3964733103 shutdown_connections 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9009a290 0x7f6d900c5f80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 --2- 172.21.15.28:0/3964733103 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d9015f260 0x7f6d900cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- 172.21.15.28:0/3964733103 >> 172.21.15.28:0/3964733103 conn(0x7f6d900b34d0 msgr2=0x7f6d900b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- 172.21.15.28:0/3964733103 shutdown_connections 2024-03-21T08:41:27.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- 172.21.15.28:0/3964733103 wait complete. 2024-03-21T08:41:27.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 Processor -- start 2024-03-21T08:41:27.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.774+0000 7f6d94a00640 1 -- start start 2024-03-21T08:41:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 0x7f6d90165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d900cd5d0 0x7f6d90164710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 0x7f6d900cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900c6e50 con 0x7f6d9015f260 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900c6cd0 con 0x7f6d9009a290 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900c6fd0 con 0x7f6d900cd5d0 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 0x7f6d90165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 0x7f6d900cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 0x7f6d90165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:58986/0 (socket says 172.21.15.28:58986) 2024-03-21T08:41:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8de00640 1 -- 172.21.15.28:0/854634744 learned_addr learned my addr 172.21.15.28:0/854634744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d87e00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d900cd5d0 0x7f6d90164710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 -- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d900cd5d0 msgr2=0x7f6d90164710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d900cd5d0 0x7f6d90164710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 -- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 msgr2=0x7f6d90165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 0x7f6d90165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 -- 172.21.15.28:0/854634744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900cb8d0 con 0x7f6d9015f260 2024-03-21T08:41:27.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d8e800640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 0x7f6d900cb390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ab000 tx=0x7f6d88019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880bc020 con 0x7f6d9015f260 2024-03-21T08:41:27.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d880b86b0 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d90164df0 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880b8aa0 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d901653b0 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9004f410 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.778+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6d880bf020 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.782+0000 7f6d86000640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 0x7f6d6c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.782+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6d88120f80 con 0x7f6d9015f260 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.782+0000 7f6d8de00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 0x7f6d6c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.782+0000 7f6d8de00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 0x7f6d6c06a010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6d9006df70 tx=0x7f6d80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.786+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6d88016620 con 0x7f6d9015f260 2024-03-21T08:41:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.986+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6d90071a30 con 0x7f6d9015f260 2024-03-21T08:41:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.990+0000 7f6d86000640 1 -- 172.21.15.28:0/854634744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+167128 (secure 0 0 0) 0x7f6d880a59c0 con 0x7f6d9015f260 2024-03-21T08:41:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 msgr2=0x7f6d6c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 0x7f6d6c06a010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6d9006df70 tx=0x7f6d80009190 comp rx=0 tx=0).stop 2024-03-21T08:41:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 msgr2=0x7f6d900cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 0x7f6d900cb390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ab000 tx=0x7f6d88019710 comp rx=0 tx=0).stop 2024-03-21T08:41:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 shutdown_connections 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015f260 0x7f6d900cb390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6d6c067b50 0x7f6d6c06a010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6d900cd5d0 0x7f6d90164710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 --2- 172.21.15.28:0/854634744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6d9009a290 0x7f6d90165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 >> 172.21.15.28:0/854634744 conn(0x7f6d900b34d0 msgr2=0x7f6d901560e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 shutdown_connections 2024-03-21T08:41:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:27.994+0000 7f6d94a00640 1 -- 172.21.15.28:0/854634744 wait complete. 2024-03-21T08:41:28.016 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":2277034983},{"type":"v1","addr":"172.21.15.151:6833","nonce":2277034983}]},"active_addr":"172.21.15.151:6833/2277034983","active_change":"2024-03-21T08:40:47.708422+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3695867403}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3875181327}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3919540698}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2710643650}]}]} 2024-03-21T08:41:28.019 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-21T08:41:28.019 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-21T08:41:28.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.158+0000 7ff27f000640 1 Processor -- start 2024-03-21T08:41:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.158+0000 7ff27f000640 1 -- start start 2024-03-21T08:41:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2780cd380 0x7ff2780b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27815bf00 con 0x7ff2780af060 2024-03-21T08:41:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27f000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2780cdad0 con 0x7ff2780b5520 2024-03-21T08:41:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780b3a30 con 0x7ff2780cd380 2024-03-21T08:41:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff277e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2780af060 0x7ff2780b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36794/0 (socket says 172.21.15.28:36794) 2024-03-21T08:41:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff277e00640 1 -- 172.21.15.28:0/3966069006 learned_addr learned my addr 172.21.15.28:0/3966069006 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27d000640 1 --2- 172.21.15.28:0/3966069006 >> v2:172.21.15.28:3300/0 conn(0x7ff2780cd380 0x7ff2780b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27d000640 1 -- 172.21.15.28:0/3966069006 >> v1:172.21.15.151:6789/0 conn(0x7ff2780b5520 legacy=0x7ff27815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff277400640 1 -- 172.21.15.28:0/3966069006 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286542824 0 0) 0x7ff27815bf00 con 0x7ff2780af060 2024-03-21T08:41:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27d000640 1 -- 172.21.15.28:0/3966069006 >> v1:172.21.15.28:6789/0 conn(0x7ff2780af060 legacy=0x7ff2780b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27d000640 1 -- 172.21.15.28:0/3966069006 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780cddb0 con 0x7ff2780cd380 2024-03-21T08:41:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.162+0000 7ff27d000640 1 --2- 172.21.15.28:0/3966069006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780cd380 0x7ff2780b34f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff270009d90 tx=0x7ff27001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8b3a92553895d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277400640 1 -- 172.21.15.28:0/3966069006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700b2070 con 0x7ff2780cd380 2024-03-21T08:41:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277400640 1 -- 172.21.15.28:0/3966069006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2700a5890 con 0x7ff2780cd380 2024-03-21T08:41:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277400640 1 -- 172.21.15.28:0/3966069006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700a5c60 con 0x7ff2780cd380 2024-03-21T08:41:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- 172.21.15.28:0/3966069006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780cd380 msgr2=0x7ff2780b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 --2- 172.21.15.28:0/3966069006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780cd380 0x7ff2780b34f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff270009d90 tx=0x7ff27001c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- 172.21.15.28:0/3966069006 shutdown_connections 2024-03-21T08:41:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 --2- 172.21.15.28:0/3966069006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780cd380 0x7ff2780b34f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- 172.21.15.28:0/3966069006 >> 172.21.15.28:0/3966069006 conn(0x7ff2780cc230 msgr2=0x7ff2780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- 172.21.15.28:0/3966069006 shutdown_connections 2024-03-21T08:41:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- 172.21.15.28:0/3966069006 wait complete. 2024-03-21T08:41:28.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 Processor -- start 2024-03-21T08:41:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- start start 2024-03-21T08:41:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2780cd380 0x7ff278151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff278152340 0x7ff2781490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27815d2a0 con 0x7ff2780b5520 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27815d120 con 0x7ff278152340 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27f000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27815d420 con 0x7ff2780cd380 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51430/0 (socket says 172.21.15.28:51430) 2024-03-21T08:41:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277e00640 1 -- 172.21.15.28:0/1845867867 learned_addr learned my addr 172.21.15.28:0/1845867867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff27d000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff278152340 0x7ff2781490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff26f400640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2780cd380 0x7ff278151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.166+0000 7ff277e00640 1 -- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2780cd380 msgr2=0x7ff278151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff277e00640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2780cd380 0x7ff278151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff277e00640 1 -- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff278152340 msgr2=0x7ff2781490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff277e00640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff278152340 0x7ff2781490b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff277e00640 1 -- 172.21.15.28:0/1845867867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278152880 con 0x7ff2780b5520 2024-03-21T08:41:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff277e00640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff26800c4d0 tx=0x7ff26800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268018070 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2680074e0 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268004a30 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff278152b60 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff278153120 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27804f410 con 0x7ff2780b5520 2024-03-21T08:41:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff26801b070 con 0x7ff2780b5520 2024-03-21T08:41:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 0x7ff25406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff2680855e0 con 0x7ff2780b5520 2024-03-21T08:41:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.170+0000 7ff26f400640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 0x7ff25406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.174+0000 7ff26f400640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 0x7ff25406a010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff278148f70 tx=0x7ff260009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.174+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff268055530 con 0x7ff2780b5520 2024-03-21T08:41:28.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2780ccd50 con 0x7ff2780b5520 2024-03-21T08:41:28.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff274c00640 1 -- 172.21.15.28:0/1845867867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7ff2680595d0 con 0x7ff2780b5520 2024-03-21T08:41:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:28.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:28.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 msgr2=0x7ff25406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 0x7ff25406a010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff278148f70 tx=0x7ff260009040 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 msgr2=0x7ff2781448f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff26800c4d0 tx=0x7ff26800c990 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 shutdown_connections 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff278152340 0x7ff2781490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff254067b50 0x7ff25406a010 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2780cd380 0x7ff278151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 --2- 172.21.15.28:0/1845867867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2780b5520 0x7ff2781448f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 >> 172.21.15.28:0/1845867867 conn(0x7ff2780cc230 msgr2=0x7ff278150a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 shutdown_connections 2024-03-21T08:41:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.346+0000 7ff27f000640 1 -- 172.21.15.28:0/1845867867 wait complete. 2024-03-21T08:41:28.371 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-21T08:41:28.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:28.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 Processor -- start 2024-03-21T08:41:28.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 -- start start 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8a00d0480 0x7fb8a0099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00cd1d0 con 0x7fb8a0099a30 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a00ccf90 con 0x7fb8a00d0480 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a7000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00cd4b0 con 0x7fb8a015f220 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a6600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8a00d0480 0x7fb8a0099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a5200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb8a0099a30 0x7fb8a00c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36802/0 (socket says 172.21.15.28:36802) 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a5200640 1 -- 172.21.15.28:0/2135620696 learned_addr learned my addr 172.21.15.28:0/2135620696 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:28.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a5c00640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a6600640 1 -- 172.21.15.28:0/2135620696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 msgr2=0x7fb8a00b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a6600640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a6600640 1 -- 172.21.15.28:0/2135620696 >> v1:172.21.15.28:6789/0 conn(0x7fb8a0099a30 legacy=0x7fb8a00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb8a6600640 1 -- 172.21.15.28:0/2135620696 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00cd790 con 0x7fb8a00d0480 2024-03-21T08:41:28.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.510+0000 7fb89fe00640 1 -- 172.21.15.28:0/2135620696 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492136505 0 0) 0x7fb8a00cd1d0 con 0x7fb8a0099a30 2024-03-21T08:41:28.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a6600640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a00d0480 0x7fb8a0099eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb898009d90 tx=0x7fb89801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1242350b28484a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb89fe00640 1 -- 172.21.15.28:0/2135620696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980b2070 con 0x7fb8a00d0480 2024-03-21T08:41:28.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb89fe00640 1 -- 172.21.15.28:0/2135620696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8980a5830 con 0x7fb8a00d0480 2024-03-21T08:41:28.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb89fe00640 1 -- 172.21.15.28:0/2135620696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980a5c00 con 0x7fb8a00d0480 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 -- 172.21.15.28:0/2135620696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a00d0480 msgr2=0x7fb8a0099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a00d0480 0x7fb8a0099eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb898009d90 tx=0x7fb89801c970 comp rx=0 tx=0).stop 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 -- 172.21.15.28:0/2135620696 shutdown_connections 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a00d0480 0x7fb8a0099eb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 --2- 172.21.15.28:0/2135620696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 -- 172.21.15.28:0/2135620696 >> 172.21.15.28:0/2135620696 conn(0x7fb8a00b3390 msgr2=0x7fb8a00b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 -- 172.21.15.28:0/2135620696 shutdown_connections 2024-03-21T08:41:28.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.514+0000 7fb8a7000640 1 -- 172.21.15.28:0/2135620696 wait complete. 2024-03-21T08:41:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 Processor -- start 2024-03-21T08:41:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- start start 2024-03-21T08:41:28.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8a00d0480 0x7fb8a0166060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00c6310 con 0x7fb8a0099a30 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00c6490 con 0x7fb8a00d0480 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00c6610 con 0x7fb8a015f220 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51454/0 (socket says 172.21.15.28:51454) 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 -- 172.21.15.28:0/3909805685 learned_addr learned my addr 172.21.15.28:0/3909805685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a6600640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5200640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8a00d0480 0x7fb8a0166060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 -- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 msgr2=0x7fb8a00b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 -- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8a00d0480 msgr2=0x7fb8a0166060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8a00d0480 0x7fb8a0166060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 -- 172.21.15.28:0/3909805685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0164f20 con 0x7fb8a0099a30 2024-03-21T08:41:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a5c00640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb89400e930 tx=0x7fb89400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894017070 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400ca00 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a0165200 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89400cdb0 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a01665a0 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.518+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0075270 con 0x7fb8a0099a30 2024-03-21T08:41:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.522+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb89401a070 con 0x7fb8a0099a30 2024-03-21T08:41:28.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.522+0000 7fb89d600640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 0x7fb880069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.522+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb894086650 con 0x7fb8a0099a30 2024-03-21T08:41:28.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.522+0000 7fb8a5200640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 0x7fb880069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.522+0000 7fb8a5200640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 0x7fb880069fe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8a009ae70 tx=0x7fb88c007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.526+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb8940566f0 con 0x7fb8a0099a30 2024-03-21T08:41:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.690+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a00c8200 con 0x7fb8a0099a30 2024-03-21T08:41:28.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.690+0000 7fb89d600640 1 -- 172.21.15.28:0/3909805685 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7fb89405a790 con 0x7fb8a0099a30 2024-03-21T08:41:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:28.694 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:28.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 msgr2=0x7fb880069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 0x7fb880069fe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8a009ae70 tx=0x7fb88c007df0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 msgr2=0x7fb8a0147c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb89400e930 tx=0x7fb89400edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 shutdown_connections 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb880067b20 0x7fb880069fe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb8a015f220 0x7fb8a00b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb8a00d0480 0x7fb8a0166060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 --2- 172.21.15.28:0/3909805685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0099a30 0x7fb8a0147c00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 >> 172.21.15.28:0/3909805685 conn(0x7fb8a00b3390 msgr2=0x7fb8a00c7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 shutdown_connections 2024-03-21T08:41:28.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.694+0000 7fb8a7000640 1 -- 172.21.15.28:0/3909805685 wait complete. 2024-03-21T08:41:28.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-21T08:41:28.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-21T08:41:28.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-21T08:41:28.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-21T08:41:28.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-21T08:41:28.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-21T08:41:28.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-21T08:41:28.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-21T08:41:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.962+0000 7f73bf600640 1 Processor -- start 2024-03-21T08:41:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.962+0000 7f73bf600640 1 -- start start 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bf600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f73b8161250 0x7f73b80b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bf600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cfc00 0x7f73b80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bf600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b80d0220 con 0x7f73b80cc0b0 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bf600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b8161630 con 0x7f73b8161250 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bf600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806c950 con 0x7f73b80cfc00 2024-03-21T08:41:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 Processor -- start 2024-03-21T08:41:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f73b8161250 0x7f73b80b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f73b8161250 0x7f73b80b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59000/0 (socket says 172.21.15.28:59000) 2024-03-21T08:41:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 -- 172.21.15.28:0/1484406389 learned_addr learned my addr 172.21.15.28:0/1484406389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f3f3dc00640 1 Processor -- start 2024-03-21T08:41:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f3f3dc00640 1 -- start start 2024-03-21T08:41:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f9343200640 1 Processor -- start 2024-03-21T08:41:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 -- start start 2024-03-21T08:41:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f9343200640 1 -- start start 2024-03-21T08:41:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 -- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cfc00 msgr2=0x7f73b80b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8415fba0 con 0x7fac8415bd20 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8415fe80 con 0x7fac840b0e80 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac8ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8408bfd0 con 0x7fac840cebd0 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac83400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fac840b0e80 0x7fac840ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39690/0 (socket says 172.21.15.28:39690) 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac83400640 1 -- 172.21.15.28:0/585144652 learned_addr learned my addr 172.21.15.28:0/585144652 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7fac88c00640 1 --2- 172.21.15.28:0/585144652 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 --2- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cfc00 0x7f73b80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 -- 172.21.15.28:0/1484406389 >> v1:172.21.15.28:6789/0 conn(0x7f73b80cc0b0 legacy=0x7f73b80cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.966+0000 7f73bcc00640 1 -- 172.21.15.28:0/1484406389 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b806cc30 con 0x7f73b8161250 2024-03-21T08:41:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9343200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f933c15e080 0x7f933c164ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9343200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0cf140 con 0x7f933c0b2ad0 2024-03-21T08:41:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9343200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0cf420 con 0x7f933c154350 2024-03-21T08:41:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9343200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c06df00 con 0x7f933c15e080 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f3dc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 0x7f3f38071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f3dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f3dc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3806ea00 con 0x7f3f38162c90 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f3dc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f380b5000 con 0x7f3f380b4890 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f3dc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3806dc40 con 0x7f3f38071fc0 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f37400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f37400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37208/0 (socket says 172.21.15.28:37208) 2024-03-21T08:41:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f37400640 1 -- 172.21.15.28:0/3735222074 learned_addr learned my addr 172.21.15.28:0/3735222074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9341200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f933c15e080 0x7f933c164ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f933be00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f933c154350 0x7f933c0cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39692/0 (socket says 172.21.15.28:39692) 2024-03-21T08:41:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f933be00640 1 -- 172.21.15.28:0/2805101712 learned_addr learned my addr 172.21.15.28:0/2805101712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7fac82a00640 1 -- 172.21.15.28:0/585144652 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068495056 0 0) 0x7fac8415fe80 con 0x7fac840b0e80 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7fac82a00640 1 -- 172.21.15.28:0/585144652 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac68003530 con 0x7fac840b0e80 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7fac88c00640 1 -- 172.21.15.28:0/585144652 >> v1:172.21.15.151:6789/0 conn(0x7fac840b0e80 legacy=0x7fac840ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7fac88c00640 1 -- 172.21.15.28:0/585144652 >> v1:172.21.15.28:6789/0 conn(0x7fac8415bd20 legacy=0x7fac840b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7fac88c00640 1 -- 172.21.15.28:0/585144652 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84160160 con 0x7fac840cebd0 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9341200640 1 -- 172.21.15.28:0/2805101712 >> v1:172.21.15.151:6789/0 conn(0x7f933c154350 legacy=0x7f933c0cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9341200640 1 -- 172.21.15.28:0/2805101712 >> v1:172.21.15.28:6789/0 conn(0x7f933c0b2ad0 legacy=0x7f933c15d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f9341200640 1 -- 172.21.15.28:0/2805101712 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c06e1e0 con 0x7f933c15e080 2024-03-21T08:41:28.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306386594 0 0) 0x7f3f3806ea00 con 0x7f3f38162c90 2024-03-21T08:41:28.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c003530 con 0x7f3f38162c90 2024-03-21T08:41:28.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f36000640 1 --2- 172.21.15.28:0/3735222074 >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 0x7f3f38071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3c1fc00640 1 Processor -- start 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f558b200640 1 Processor -- start 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f558b200640 1 -- start start 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3f37400640 1 -- 172.21.15.28:0/3735222074 >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 msgr2=0x7f3f38071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3f37400640 1 --2- 172.21.15.28:0/3735222074 >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 0x7f3f38071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3f37400640 1 -- 172.21.15.28:0/3735222074 >> v1:172.21.15.151:6789/0 conn(0x7f3f38162c90 legacy=0x7f3f38163070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3f37400640 1 -- 172.21.15.28:0/3735222074 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3806df20 con 0x7f3f38071fc0 2024-03-21T08:41:28.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2477567558 0 0) 0x7f3f1c003530 con 0x7f3f38162c90 2024-03-21T08:41:28.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3f36000640 1 --2- 172.21.15.28:0/3735222074 >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 0x7f3f38071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:28.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.970+0000 7f3c1fc00640 1 -- start start 2024-03-21T08:41:28.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bcc00640 1 --2- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b8161250 0x7f73b80b33a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3bd1c285a2f166 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73b7400640 1 -- 172.21.15.28:0/1484406389 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40b0070 con 0x7f73b8161250 2024-03-21T08:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73b7400640 1 -- 172.21.15.28:0/1484406389 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a40ab600 con 0x7f73b8161250 2024-03-21T08:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73b7400640 1 -- 172.21.15.28:0/1484406389 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40ab9d0 con 0x7f73b8161250 2024-03-21T08:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c1815df80 0x7f3c180d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 0x7f3c180d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 0x7f3c180d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac88c00640 1 --2- 172.21.15.28:0/585144652 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84166d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009dc0 tx=0x7fac7c01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbc879c07e7dc79 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c180d32f0 con 0x7f3c1815df80 2024-03-21T08:41:28.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c180d26e0 con 0x7f3c180d3780 2024-03-21T08:41:28.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558b200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 0x7f558003e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5580007810 0x7f5580054080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 -- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b8161250 msgr2=0x7f73b80b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 --2- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b8161250 0x7f73b80b33a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a401c4b0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac82a00640 1 -- 172.21.15.28:0/585144652 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0b2070 con 0x7fac840cebd0 2024-03-21T08:41:28.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac82a00640 1 -- 172.21.15.28:0/585144652 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c0a5790 con 0x7fac840cebd0 2024-03-21T08:41:28.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac82a00640 1 -- 172.21.15.28:0/585144652 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0a5b60 con 0x7fac840cebd0 2024-03-21T08:41:28.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 -- 172.21.15.28:0/1484406389 shutdown_connections 2024-03-21T08:41:28.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 --2- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cfc00 0x7f73b80b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c17e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 0x7f3c180d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 --2- 172.21.15.28:0/1484406389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b8161250 0x7f73b80b33a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f73bf600640 1 -- 172.21.15.28:0/1484406389 >> 172.21.15.28:0/1484406389 conn(0x7f73b80ce3a0 msgr2=0x7f73b80ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c17e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 0x7f3c180d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59014/0 (socket says 172.21.15.28:59014) 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c17e00640 1 -- 172.21.15.28:0/2580698749 learned_addr learned my addr 172.21.15.28:0/2580698749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180d29c0 con 0x7f3c180b9630 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2580698749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 0x7f3c180d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1d200640 1 --2- 172.21.15.28:0/2580698749 >> v2:172.21.15.28:3300/0 conn(0x7f3c1815df80 0x7f3c180d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558b200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5580057df0 con 0x7f5580053c20 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558003bfd0 con 0x7f5580007810 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558b200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55800545c0 con 0x7f55800e32e0 2024-03-21T08:41:28.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5580007810 0x7f5580054080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5580007810 0x7f5580054080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51492/0 (socket says 172.21.15.28:51492) 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f558a800640 1 -- 172.21.15.28:0/2812353500 learned_addr learned my addr 172.21.15.28:0/2812353500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac8ac00640 1 -- 172.21.15.28:0/585144652 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 msgr2=0x7fac84166d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7fac8ac00640 1 --2- 172.21.15.28:0/585144652 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84166d00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009dc0 tx=0x7fac7c01c910 comp rx=0 tx=0).stop 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2580698749 >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 msgr2=0x7f3c180d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2580698749 >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 0x7f3c180d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2580698749 >> v2:172.21.15.28:3300/0 conn(0x7f3c1815df80 msgr2=0x7f3c180d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2580698749 >> v2:172.21.15.28:3300/0 conn(0x7f3c1815df80 0x7f3c180d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2580698749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c180d2ca0 con 0x7f3c180b9630 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f5589e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 0x7f558003e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f9341200640 1 --2- 172.21.15.28:0/2805101712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c15e080 0x7f933c164ad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9334009d90 tx=0x7f933401d3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ce942a6d8b4c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.974+0000 7f933aa00640 1 -- 172.21.15.28:0/2805101712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340b2070 con 0x7f933c15e080 2024-03-21T08:41:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f933aa00640 1 -- 172.21.15.28:0/2805101712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93340a5940 con 0x7f933c15e080 2024-03-21T08:41:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f933aa00640 1 -- 172.21.15.28:0/2805101712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340a5d10 con 0x7f933c15e080 2024-03-21T08:41:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f5588a00640 1 -- 172.21.15.28:0/2812353500 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20669888 0 0) 0x7f5580057df0 con 0x7f5580053c20 2024-03-21T08:41:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7fac8ac00640 1 -- 172.21.15.28:0/585144652 shutdown_connections 2024-03-21T08:41:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7fac8ac00640 1 --2- 172.21.15.28:0/585144652 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84166d00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7fac8ac00640 1 -- 172.21.15.28:0/585144652 >> 172.21.15.28:0/585144652 conn(0x7fac840d1ab0 msgr2=0x7fac840d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f5588a00640 1 -- 172.21.15.28:0/2812353500 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c003530 con 0x7f5580053c20 2024-03-21T08:41:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 Processor -- start 2024-03-21T08:41:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f37400640 1 --2- 172.21.15.28:0/3735222074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009870 tx=0x7f3f2401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5ff483cbe001d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 -- 172.21.15.28:0/2805101712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c15e080 msgr2=0x7f933c164ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 --2- 172.21.15.28:0/2805101712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c15e080 0x7f933c164ad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9334009d90 tx=0x7f933401d3f0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 -- start start 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f558a800640 1 -- 172.21.15.28:0/2812353500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 msgr2=0x7f558003e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f558a800640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 0x7f558003e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f558a800640 1 -- 172.21.15.28:0/2812353500 >> v1:172.21.15.151:6789/0 conn(0x7f5580053c20 legacy=0x7f55800e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f558a800640 1 -- 172.21.15.28:0/2812353500 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55800580d0 con 0x7f5580007810 2024-03-21T08:41:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f5589e00640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 0x7f558003e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f240b0070 con 0x7f3f38071fc0 2024-03-21T08:41:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f240a4e50 con 0x7f3f38071fc0 2024-03-21T08:41:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f35600640 1 -- 172.21.15.28:0/3735222074 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f240b0070 con 0x7f3f38071fc0 2024-03-21T08:41:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7fac8ac00640 1 -- 172.21.15.28:0/585144652 shutdown_connections 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7fac8ac00640 1 -- 172.21.15.28:0/585144652 wait complete. 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 -- 172.21.15.28:0/2805101712 shutdown_connections 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 --2- 172.21.15.28:0/2805101712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c15e080 0x7f933c164ad0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 -- 172.21.15.28:0/2805101712 >> 172.21.15.28:0/2805101712 conn(0x7f933c06b770 msgr2=0x7f933c0b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f73bf600640 1 -- 172.21.15.28:0/1484406389 shutdown_connections 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f73bf600640 1 -- 172.21.15.28:0/1484406389 wait complete. 2024-03-21T08:41:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f73bf600640 1 Processor -- start 2024-03-21T08:41:28.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 -- 172.21.15.28:0/2805101712 shutdown_connections 2024-03-21T08:41:28.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f73bf600640 1 -- start start 2024-03-21T08:41:28.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3735222074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 msgr2=0x7f3f380b2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3735222074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009870 tx=0x7f3f2401c480 comp rx=0 tx=0).stop 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2658140d40 0x7f26580b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26580cd040 con 0x7f26580cc680 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26580b2b70 con 0x7f2658140d40 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f265de00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26580b2e50 con 0x7f2658143990 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f2656a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f2656a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37244/0 (socket says 172.21.15.28:37244) 2024-03-21T08:41:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f2656a00640 1 -- 172.21.15.28:0/2249781437 learned_addr learned my addr 172.21.15.28:0/2249781437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2656000640 1 --2- 172.21.15.28:0/2249781437 >> v2:172.21.15.151:3300/0 conn(0x7f2658140d40 0x7f26580b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2580698749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 0x7f3c180d08c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3c180d1800 tx=0x7f3c1001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653267d18bb3e71b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3735222074 shutdown_connections 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3735222074 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f38071fc0 0x7f3f380b2ad0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3735222074 >> v2:172.21.15.151:3300/0 conn(0x7f3f380b4890 0x7f3f38071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3735222074 >> 172.21.15.28:0/3735222074 conn(0x7f3f380674f0 msgr2=0x7f3f3806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3735222074 shutdown_connections 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3735222074 wait complete. 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2655600640 1 -- 172.21.15.28:0/2249781437 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023015280 0 0) 0x7f26580cd040 con 0x7f26580cc680 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2656a00640 1 -- 172.21.15.28:0/2249781437 >> v2:172.21.15.151:3300/0 conn(0x7f2658140d40 msgr2=0x7f26580b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2656a00640 1 --2- 172.21.15.28:0/2249781437 >> v2:172.21.15.151:3300/0 conn(0x7f2658140d40 0x7f26580b1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:28.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2656a00640 1 -- 172.21.15.28:0/2249781437 >> v1:172.21.15.151:6789/0 conn(0x7f26580cc680 legacy=0x7f265815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f2656a00640 1 -- 172.21.15.28:0/2249781437 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26580b3130 con 0x7f2658143990 2024-03-21T08:41:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c17400640 1 -- 172.21.15.28:0/2580698749 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100b0070 con 0x7f3c180b9630 2024-03-21T08:41:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c17400640 1 -- 172.21.15.28:0/2580698749 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c100ab600 con 0x7f3c180b9630 2024-03-21T08:41:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 Processor -- start 2024-03-21T08:41:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 0x7f73b8154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.978+0000 7f9343200640 1 -- 172.21.15.28:0/2805101712 wait complete. 2024-03-21T08:41:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 -- start start 2024-03-21T08:41:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f9343200640 1 Processor -- start 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b80cfc00 0x7f73b8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 0x7f73b8154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73b8161250 0x7f73b8155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806dfd0 con 0x7f73b8161250 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806de50 con 0x7f73b80cfc00 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bf600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806e150 con 0x7f73b80cc0b0 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 0x7f73b8154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37250/0 (socket says 172.21.15.28:37250) 2024-03-21T08:41:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 -- 172.21.15.28:0/585527761 learned_addr learned my addr 172.21.15.28:0/585527761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f5755200640 1 Processor -- start 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f5755200640 1 -- start start 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 msgr2=0x7f3c180d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 --2- 172.21.15.28:0/2580698749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 0x7f3c180d08c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3c180d1800 tx=0x7f3c1001c480 comp rx=0 tx=0).stop 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 shutdown_connections 2024-03-21T08:41:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 --2- 172.21.15.28:0/2580698749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c180b9630 0x7f3c180d08c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 --2- 172.21.15.28:0/2580698749 >> v2:172.21.15.151:3300/0 conn(0x7f3c180d3780 0x7f3c180d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 --2- 172.21.15.28:0/2580698749 >> v2:172.21.15.28:3300/0 conn(0x7f3c1815df80 0x7f3c180d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 >> 172.21.15.28:0/2580698749 conn(0x7f3c1806c8e0 msgr2=0x7f3c1806ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f9343200640 1 -- start start 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 shutdown_connections 2024-03-21T08:41:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2580698749 wait complete. 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73b7e00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b80cfc00 0x7f73b8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bd600640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73b8161250 0x7f73b8155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 -- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b80cfc00 msgr2=0x7f73b8154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f73b80cfc00 0x7f73b8154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 -- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73b8161250 msgr2=0x7f73b8155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73b8161250 0x7f73b8155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b80b7430 con 0x7f73b80cc0b0 2024-03-21T08:41:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f73bcc00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 0x7f73b8154300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f73a401ab60 tx=0x7f73a4019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f3f3dc00640 1 Processor -- start 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840b0e80 0x7fac8415c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84206f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 0x7fac8415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84169be0 con 0x7fac840b0e80 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84169a60 con 0x7fac8415bd20 2024-03-21T08:41:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac8ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac84169d60 con 0x7fac840cebd0 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac88c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 0x7fac8415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac88c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 0x7fac8415a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59050/0 (socket says 172.21.15.28:59050) 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7fac88c00640 1 -- 172.21.15.28:0/3327644614 learned_addr learned my addr 172.21.15.28:0/3327644614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.982+0000 7f5755200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5750142990 0x7f57500cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5755200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 0x7f575015e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5755200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575006dfa0 con 0x7f575006fb20 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5755200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57500b30a0 con 0x7f575006daa0 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5755200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575006e280 con 0x7f5750142990 2024-03-21T08:41:29.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5750142990 0x7f57500cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 -- start start 2024-03-21T08:41:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5750142990 0x7f57500cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59066/0 (socket says 172.21.15.28:59066) 2024-03-21T08:41:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 -- 172.21.15.28:0/3062561742 learned_addr learned my addr 172.21.15.28:0/3062561742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 -- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 msgr2=0x7fac84206f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40abbc0 con 0x7f73b80cc0b0 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a40abd60 con 0x7f73b80cc0b0 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40c2630 con 0x7f73b80cc0b0 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fac840cebd0 0x7fac84206f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 -- 172.21.15.28:0/3327644614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840b0e80 msgr2=0x7fac8415c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840b0e80 0x7fac8415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8415d190 con 0x7fac8415bd20 2024-03-21T08:41:29.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 0x7f933c141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 Processor -- start 2024-03-21T08:41:29.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73bf600640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b80b6b90 con 0x7f73b80cc0b0 2024-03-21T08:41:29.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73bf600640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b80b7150 con 0x7f73b80cc0b0 2024-03-21T08:41:29.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f933c154350 0x7f933c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 0x7f933c141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 0x7f933c141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51540/0 (socket says 172.21.15.28:51540) 2024-03-21T08:41:29.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 -- start start 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac88c00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 0x7fac8415a4d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fac7c01afc0 tx=0x7fac7c0196b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f558a800640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5580007810 0x7f5580054080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5574009870 tx=0x7f557401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310d4c73fbf82b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574e800640 1 --2- 172.21.15.28:0/3062561742 >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 0x7f575015e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 -- 172.21.15.28:0/3062561742 >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 msgr2=0x7f575015e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 --2- 172.21.15.28:0/3062561742 >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 0x7f575015e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 -- 172.21.15.28:0/3062561742 >> v1:172.21.15.28:6789/0 conn(0x7f575006fb20 legacy=0x7f575015d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f73bf600640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73b804f410 con 0x7f73b80cc0b0 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 -- 172.21.15.28:0/3428161400 learned_addr learned my addr 172.21.15.28:0/3428161400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f933c15e080 0x7f933c13aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c167bc0 con 0x7f933c0b2ad0 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c167a40 con 0x7f933c15e080 2024-03-21T08:41:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c167d40 con 0x7f933c154350 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0bd020 con 0x7fac8415bd20 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c0b3040 con 0x7fac8415bd20 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0b2070 con 0x7fac8415bd20 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574de00640 1 -- 172.21.15.28:0/3062561742 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575006e560 con 0x7f5750142990 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f574e800640 1 --2- 172.21.15.28:0/3062561742 >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 0x7f575015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f2656a00640 1 --2- 172.21.15.28:0/2249781437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2640014420 tx=0x7f2640025540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4219d6b9d45a4885 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c180b9630 0x7f3c1815ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c180d3780 0x7f3c181f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 0x7f3c181f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180b6090 con 0x7f3c180d3780 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180b5f10 con 0x7f3c180b9630 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180b6210 con 0x7f3c1815df80 2024-03-21T08:41:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8415d470 con 0x7fac8415bd20 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8415da30 con 0x7fac8415bd20 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5588a00640 1 -- 172.21.15.28:0/2812353500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740b0070 con 0x7f5580007810 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5588a00640 1 -- 172.21.15.28:0/2812353500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55740ab600 con 0x7f5580007810 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f5588a00640 1 -- 172.21.15.28:0/2812353500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740ab9d0 con 0x7f5580007810 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 0x7f3f38155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f380b4890 0x7f3f3816c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f38162c90 0x7f3f38159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 0x7f3c181f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 0x7f3c181f3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37278/0 (socket says 172.21.15.28:37278) 2024-03-21T08:41:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2990122626 learned_addr learned my addr 172.21.15.28:0/2990122626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380b39a0 con 0x7f3f38162c90 2024-03-21T08:41:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380b3820 con 0x7f3f38071fc0 2024-03-21T08:41:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f9341200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f933c15e080 0x7f933c13aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933b400640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f933c154350 0x7f933c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3f3dc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380b3b20 con 0x7f3f380b4890 2024-03-21T08:41:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c17e00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c180d3780 0x7f3c181f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 -- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f933c154350 msgr2=0x7f933c152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f933c154350 0x7f933c152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 -- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f933c15e080 msgr2=0x7f933c13aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f933c15e080 0x7f933c13aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f933be00640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c141a40 con 0x7f933c0b2ad0 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c180b9630 msgr2=0x7f3c1815ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c180b9630 0x7f3c1815ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c180d3780 msgr2=0x7f3c181f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c180d3780 0x7f3c181f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.986+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1815b270 con 0x7f3c1815df80 2024-03-21T08:41:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac8409fb00 con 0x7fac8415bd20 2024-03-21T08:41:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 0x7f3f38155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f933b400640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f933c154350 0x7f933c152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f933be00640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 0x7f933c141460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f932800c830 tx=0x7f932800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 -- 172.21.15.28:0/2812353500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5580007810 msgr2=0x7f5580054080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5580007810 0x7f5580054080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5574009870 tx=0x7f557401c560 comp rx=0 tx=0).stop 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 -- 172.21.15.28:0/2812353500 shutdown_connections 2024-03-21T08:41:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5580007810 0x7f5580054080 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 --2- 172.21.15.28:0/2812353500 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f55800e32e0 0x7f558003e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 -- 172.21.15.28:0/2812353500 >> 172.21.15.28:0/2812353500 conn(0x7f55800559f0 msgr2=0x7f5580055e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328015020 con 0x7f933c0b2ad0 2024-03-21T08:41:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9328007a90 con 0x7f933c0b2ad0 2024-03-21T08:41:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c141d20 con 0x7f933c0b2ad0 2024-03-21T08:41:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 0x7f3f38155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59090/0 (socket says 172.21.15.28:59090) 2024-03-21T08:41:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 -- 172.21.15.28:0/3591342035 learned_addr learned my addr 172.21.15.28:0/3591342035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328007e40 con 0x7f933c0b2ad0 2024-03-21T08:41:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c1422e0 con 0x7f933c0b2ad0 2024-03-21T08:41:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 -- 172.21.15.28:0/2812353500 shutdown_connections 2024-03-21T08:41:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f2655600640 1 -- 172.21.15.28:0/2249781437 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26400b5070 con 0x7f2658143990 2024-03-21T08:41:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f2655600640 1 -- 172.21.15.28:0/2249781437 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26400af420 con 0x7f2658143990 2024-03-21T08:41:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36000640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f380b4890 0x7f3f3816c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 -- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f380b4890 msgr2=0x7f3f3816c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fac7c0c0020 con 0x7fac8415bd20 2024-03-21T08:41:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3c1dc00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 0x7f3c181f3150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c1001cd20 tx=0x7f3c10019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fac64067bf0 0x7fac6406a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fac7c120d00 con 0x7fac8415bd20 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f37400640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f38162c90 0x7f3f38159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f380b4890 0x7f3f3816c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 -- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f38162c90 msgr2=0x7f3f38159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f38162c90 0x7f3f38159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36a00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38156470 con 0x7f3f38071fc0 2024-03-21T08:41:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f933c1f6180 con 0x7f933c0b2ad0 2024-03-21T08:41:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7fac64071370 0x7fac640737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac7c0acec0 con 0x7fac64071370 2024-03-21T08:41:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac7c123360 con 0x7fac8415bd20 2024-03-21T08:41:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f2655600640 1 -- 172.21.15.28:0/2249781437 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26400af7f0 con 0x7f2658143990 2024-03-21T08:41:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f558b200640 1 -- 172.21.15.28:0/2812353500 wait complete. 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7fac83e00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fac64067bf0 0x7fac6406a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f73a40cb440 con 0x7f73b80cc0b0 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7398067c20 0x7f739806a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f73a41288c0 con 0x7f73b80cc0b0 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b7e00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7398067c20 0x7f739806a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f7398071310 0x7f7398073750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7398073e20 con 0x7f7398071310 2024-03-21T08:41:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73a40f4a80 con 0x7f73b80cc0b0 2024-03-21T08:41:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f574de00640 1 --2- 172.21.15.28:0/3062561742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5750142990 0x7f57500cc540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5750067980 tx=0x7f574801c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=804ff7c082e76e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f36000640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f380b4890 0x7f3f3816c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3f37400640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f38162c90 0x7f3f38159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f73bd600640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f7398071310 0x7f7398073750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f36a00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 0x7f3f38155be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000cc80 tx=0x7f3f3000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.990+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100b0040 con 0x7f3c1815df80 2024-03-21T08:41:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c100bb070 con 0x7f3c1815df80 2024-03-21T08:41:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100aadc0 con 0x7f3c1815df80 2024-03-21T08:41:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7fac83400640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7fac64071370 0x7fac640737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f574ca00640 1 -- 172.21.15.28:0/3062561742 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480a57d0 con 0x7f5750142990 2024-03-21T08:41:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f574ca00640 1 -- 172.21.15.28:0/3062561742 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57480a5970 con 0x7f5750142990 2024-03-21T08:41:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f574ca00640 1 -- 172.21.15.28:0/3062561742 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57480a5d40 con 0x7f5750142990 2024-03-21T08:41:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c181f3690 con 0x7f3c1815df80 2024-03-21T08:41:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7fac83e00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fac64067bf0 0x7fac6406a0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac74009650 tx=0x7fac74006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f73b7e00640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7398067c20 0x7f739806a0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f73b80b3a20 tx=0x7f73a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9328018020 con 0x7f933c0b2ad0 2024-03-21T08:41:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30016020 con 0x7f3f38071fc0 2024-03-21T08:41:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f30012420 con 0x7f3f38071fc0 2024-03-21T08:41:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30012850 con 0x7f3f38071fc0 2024-03-21T08:41:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f558b200640 1 Processor -- start 2024-03-21T08:41:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7fac83400640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7fac64071370 0x7fac640737b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 -- 172.21.15.28:0/3062561742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5750142990 msgr2=0x7f57500cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 --2- 172.21.15.28:0/3062561742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5750142990 0x7f57500cc540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5750067980 tx=0x7f574801c9f0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 -- 172.21.15.28:0/2249781437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 msgr2=0x7f2658142d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 --2- 172.21.15.28:0/2249781437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2640014420 tx=0x7f2640025540 comp rx=0 tx=0).stop 2024-03-21T08:41:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f73bd600640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f7398071310 0x7f7398073750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f38156750 con 0x7f3f38071fc0 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f38151fd0 con 0x7f3f38071fc0 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f931c067c20 0x7f931c06a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9328086570 con 0x7f933c0b2ad0 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f931c071320 0x7f931c073760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f931c073e30 con 0x7f931c071320 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f933b400640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f931c067c20 0x7f931c06a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3c1fc00640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c181f3a40 con 0x7f3c1815df80 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3bec001040 con 0x7f3c1815df80 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f558b200640 1 -- start start 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 -- 172.21.15.28:0/3062561742 shutdown_connections 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 --2- 172.21.15.28:0/3062561742 >> v2:172.21.15.28:3300/0 conn(0x7f575006daa0 0x7f575015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 --2- 172.21.15.28:0/3062561742 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5750142990 0x7f57500cc540 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 -- 172.21.15.28:0/3062561742 >> 172.21.15.28:0/3062561742 conn(0x7f57500674f0 msgr2=0x7f575006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 -- 172.21.15.28:0/3062561742 shutdown_connections 2024-03-21T08:41:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 -- 172.21.15.28:0/2249781437 shutdown_connections 2024-03-21T08:41:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 --2- 172.21.15.28:0/2249781437 >> v2:172.21.15.151:3300/0 conn(0x7f2658140d40 0x7f26580b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 --2- 172.21.15.28:0/2249781437 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2658143990 0x7f2658142d20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 -- 172.21.15.28:0/2249781437 >> 172.21.15.28:0/2249781437 conn(0x7f265806b770 msgr2=0x7f26580b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== osd.5 v2:172.21.15.151:6800/2309128518 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fac7c0acec0 con 0x7fac64071370 2024-03-21T08:41:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 -- 172.21.15.28:0/2249781437 shutdown_connections 2024-03-21T08:41:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f5755200640 1 -- 172.21.15.28:0/3062561742 wait complete. 2024-03-21T08:41:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f932808d050 con 0x7f933c0b2ad0 2024-03-21T08:41:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== osd.0 v2:172.21.15.28:6808/1471116164 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7398073e20 con 0x7f7398071310 2024-03-21T08:41:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f933b400640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f931c067c20 0x7f931c06a0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f933c159e40 tx=0x7f932c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f265de00640 1 -- 172.21.15.28:0/2249781437 wait complete. 2024-03-21T08:41:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5580007810 0x7f55800dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5580053c20 0x7f55800db510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5755200640 1 Processor -- start 2024-03-21T08:41:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.994+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f381523d0 con 0x7f3f38071fc0 2024-03-21T08:41:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55800dba50 0x7f558004d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5580007810 0x7f55800dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f9341200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f931c071320 0x7f931c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5580007810 0x7f55800dafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59100/0 (socket says 172.21.15.28:59100) 2024-03-21T08:41:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589e00640 1 -- 172.21.15.28:0/4209491968 learned_addr learned my addr 172.21.15.28:0/4209491968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558003ef10 con 0x7f55800dba50 2024-03-21T08:41:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558003ed90 con 0x7f5580007810 2024-03-21T08:41:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5755200640 1 -- start start 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558a800640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55800dba50 0x7f558004d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5580053c20 0x7f55800db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55800efd00 con 0x7f5580053c20 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3f30019020 con 0x7f3f38071fc0 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3c100b3070 con 0x7f3c1815df80 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3c14c00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bf8067bf0 0x7f3bf806a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3c10120e40 con 0x7f3c1815df80 2024-03-21T08:41:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3c1d200640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bf8067bf0 0x7f3bf806a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 -- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5580007810 msgr2=0x7f55800dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5580007810 0x7f55800dafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 -- 172.21.15.28:0/4209491968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55800dba50 msgr2=0x7f558004d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55800dba50 0x7f558004d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f9341200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f931c071320 0x7f931c073760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f5589400640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55800e5410 con 0x7f5580053c20 2024-03-21T08:41:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f558a800640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55800dba50 0x7f558004d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f2ea00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f18067b70 0x7f3f1806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3f30085be0 con 0x7f3f38071fc0 2024-03-21T08:41:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:28.998+0000 7f3c14c00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7f3bf8071370 0x7f3bf80737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c100c6e50 con 0x7f3bf8071370 2024-03-21T08:41:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5589400640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5580053c20 0x7f55800db510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f557800c420 tx=0x7f557800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c10121230 con 0x7f3c1815df80 2024-03-21T08:41:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f2ea00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f3f180712e0 0x7f3f18073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f30010e50 con 0x7f3f180712e0 2024-03-21T08:41:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f30085fa0 con 0x7f3f38071fc0 2024-03-21T08:41:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c1d200640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bf8067bf0 0x7f3bf806a0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c08015630 tx=0x7f3c08015150 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575006daa0 0x7f5750141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 0x7f5750157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 0x7f5750164340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c17e00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7f3bf8071370 0x7f3bf80737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f36000640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f18067b70 0x7f3f1806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f37400640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f3f180712e0 0x7f3f18073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== osd.2 v2:172.21.15.28:6815/2368333363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f931c073e30 con 0x7f931c071320 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578017070 con 0x7f5580053c20 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55780052f0 con 0x7f5580053c20 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578003840 con 0x7f5580053c20 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558004ca50 con 0x7f5580053c20 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558004c740 con 0x7f5580053c20 2024-03-21T08:41:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 0x7f5750157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 0x7f5750157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59108/0 (socket says 172.21.15.28:59108) 2024-03-21T08:41:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f265de00640 1 Processor -- start 2024-03-21T08:41:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c17e00640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7f3bf8071370 0x7f3bf80737b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 0x7f5750164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 0x7f5750164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37298/0 (socket says 172.21.15.28:37298) 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 -- 172.21.15.28:0/3782254822 learned_addr learned my addr 172.21.15.28:0/3782254822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575015e750 con 0x7f575006daa0 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575015e5d0 con 0x7f575006fb20 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5750162d00 con 0x7f5750142990 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f265de00640 1 -- start start 2024-03-21T08:41:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f36000640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f18067b70 0x7f3f1806a030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3f380cc3f0 tx=0x7f3f28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5560001040 con 0x7f5580053c20 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 -- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 msgr2=0x7f5750157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 0x7f5750157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 -- 172.21.15.28:0/3782254822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575006daa0 msgr2=0x7f5750141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575006daa0 0x7f5750141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750157a30 con 0x7f5750142990 2024-03-21T08:41:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574d400640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f575006fb20 0x7f5750157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-21T08:41:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3f37400640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f3f180712e0 0x7f3f18073720 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f574e800640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 0x7f5750164340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e4e0 tx=0x7f573c00e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== osd.3 v2:172.21.15.28:6816/2425819575 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c100c6e50 con 0x7f3bf8071370 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.002+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c017070 con 0x7f5750142990 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c013630 con 0x7f5750142990 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00f040 con 0x7f5750142990 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5750157d10 con 0x7f5750142990 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57501582d0 con 0x7f5750142990 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26580cc680 0x7f2658141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 0x7f265815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2658143990 0x7f26581573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265815e5c0 con 0x7f2658140d40 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265815e440 con 0x7f2658143990 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f265de00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26580ccad0 con 0x7f26580cc680 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57501577d0 con 0x7f5750142990 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 0x7f265815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 0x7f265815f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51588/0 (socket says 172.21.15.28:51588) 2024-03-21T08:41:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 -- 172.21.15.28:0/2279723994 learned_addr learned my addr 172.21.15.28:0/2279723994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2657400640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2658143990 0x7f26581573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656a00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26580cc680 0x7f2658141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== osd.1 v2:172.21.15.28:6800/4044131930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f30010e50 con 0x7f3f180712e0 2024-03-21T08:41:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f557801a070 con 0x7f5580053c20 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 -- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26580cc680 msgr2=0x7f2658141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26580cc680 0x7f2658141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 -- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2658143990 msgr2=0x7f26581573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2658143990 0x7f26581573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f2656000640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265813efd0 con 0x7f2658140d40 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f557e000640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5568067ba0 0x7f556806a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f55780853b0 con 0x7f5580053c20 2024-03-21T08:41:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f557e000640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7f55680712a0 0x7f55680736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.006+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5578016ec0 con 0x7f55680712a0 2024-03-21T08:41:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f2656000640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 0x7f265815f0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f264800e930 tx=0x7f264800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5589e00640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5568067ba0 0x7f556806a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f558a800640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7f55680712a0 0x7f55680736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648017070 con 0x7f2658140d40 2024-03-21T08:41:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264800ca00 con 0x7f2658140d40 2024-03-21T08:41:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264800cdb0 con 0x7f2658140d40 2024-03-21T08:41:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f573c01a070 con 0x7f5750142990 2024-03-21T08:41:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f557808c050 con 0x7f5580053c20 2024-03-21T08:41:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5589e00640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5568067ba0 0x7f556806a060 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5584050db0 tx=0x7f5584068e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f558a800640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7f55680712a0 0x7f55680736e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265813f550 con 0x7f2658140d40 2024-03-21T08:41:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265813fb10 con 0x7f2658140d40 2024-03-21T08:41:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5730067bf0 0x7f573006a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f573c085d40 con 0x7f5750142990 2024-03-21T08:41:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5730071370 0x7f57300737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f573c00bf40 con 0x7f5730071370 2024-03-21T08:41:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f573c08a030 con 0x7f5750142990 2024-03-21T08:41:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f574de00640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5730067bf0 0x7f573006a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== osd.4 v2:172.21.15.151:6809/3622853820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5578016ec0 con 0x7f55680712a0 2024-03-21T08:41:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f574d400640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5730071370 0x7f57300737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2658073430 con 0x7f2658140d40 2024-03-21T08:41:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.010+0000 7f574d400640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5730071370 0x7f57300737b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.014+0000 7f574de00640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5730067bf0 0x7f573006a0b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f57480ae000 tx=0x7f5748004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.014+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== osd.6 v2:172.21.15.151:6808/3306491257 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f573c00bf40 con 0x7f5730071370 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.014+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f264801a070 con 0x7f2658140d40 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2634067b70 0x7f263406a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f26480546f0 con 0x7f2658140d40 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f2656a00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2634067b70 0x7f263406a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7f26340712f0 0x7f2634073730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73bf600640 1 -- 172.21.15.28:0/585527761 --> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f73b80b78a0 con 0x7f7398071310 2024-03-21T08:41:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2634073e00 con 0x7f26340712f0 2024-03-21T08:41:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26480528f0 con 0x7f2658140d40 2024-03-21T08:41:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f2657400640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7f26340712f0 0x7f2634073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f2656a00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2634067b70 0x7f263406a030 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f26400b7000 tx=0x7f26400b0000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73b4c00640 1 -- 172.21.15.28:0/585527761 <== osd.0 v2:172.21.15.28:6808/1471116164 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f73b80b78a0 con 0x7f7398071310 2024-03-21T08:41:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f2657400640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7f26340712f0 0x7f2634073730 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f7398071310 msgr2=0x7f7398073750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f7398071310 0x7f7398073750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== osd.7 v2:172.21.15.151:6824/1764908544 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2634073e00 con 0x7f26340712f0 2024-03-21T08:41:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7398067c20 msgr2=0x7f739806a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7398067c20 0x7f739806a0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f73b80b3a20 tx=0x7f73a8009040 comp rx=0 tx=0).stop 2024-03-21T08:41:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 msgr2=0x7f73b8154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 --2- 172.21.15.28:0/585527761 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f73b80cc0b0 0x7f73b8154300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f73a401ab60 tx=0x7f73a4019250 comp rx=0 tx=0).stop 2024-03-21T08:41:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73bd600640 1 -- 172.21.15.28:0/585527761 reap_dead start 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 shutdown_connections 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.018+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 >> 172.21.15.28:0/585527761 conn(0x7f73b80ce3a0 msgr2=0x7f73b80b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.022+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 shutdown_connections 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.022+0000 7f73ae000640 1 -- 172.21.15.28:0/585527761 wait complete. 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 --> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac84167020 con 0x7fac64071370 2024-03-21T08:41:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac7be00640 1 -- 172.21.15.28:0/3327644614 <== osd.5 v2:172.21.15.151:6800/2309128518 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac84167020 con 0x7fac64071370 2024-03-21T08:41:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7fac64071370 msgr2=0x7fac640737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7fac64071370 0x7fac640737b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fac64067bf0 msgr2=0x7fac6406a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fac64067bf0 0x7fac6406a0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac74009650 tx=0x7fac74006bb0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 msgr2=0x7fac8415a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.026+0000 7fac8ac00640 1 --2- 172.21.15.28:0/3327644614 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fac8415bd20 0x7fac8415a4d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fac7c01afc0 tx=0x7fac7c0196b0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7fac88c00640 1 -- 172.21.15.28:0/3327644614 reap_dead start 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 shutdown_connections 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 >> 172.21.15.28:0/3327644614 conn(0x7fac840d1ab0 msgr2=0x7fac840ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 shutdown_connections 2024-03-21T08:41:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7fac8ac00640 1 -- 172.21.15.28:0/3327644614 wait complete. 2024-03-21T08:41:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 --> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f933c142760 con 0x7f931c071320 2024-03-21T08:41:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.030+0000 7f9333e00640 1 -- 172.21.15.28:0/3428161400 <== osd.2 v2:172.21.15.28:6815/2368333363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f933c142760 con 0x7f931c071320 2024-03-21T08:41:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f931c071320 msgr2=0x7f931c073760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f931c071320 0x7f931c073760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f931c067c20 msgr2=0x7f931c06a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f931c067c20 0x7f931c06a0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f933c159e40 tx=0x7f932c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 msgr2=0x7f933c141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 --2- 172.21.15.28:0/3428161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933c0b2ad0 0x7f933c141460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f932800c830 tx=0x7f932800ccf0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9341200640 1 -- 172.21.15.28:0/3428161400 reap_dead start 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 shutdown_connections 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.034+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 >> 172.21.15.28:0/3428161400 conn(0x7f933c06b770 msgr2=0x7f933c0b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.038+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 shutdown_connections 2024-03-21T08:41:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.038+0000 7f9343200640 1 -- 172.21.15.28:0/3428161400 wait complete. 2024-03-21T08:41:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.038+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 --> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3bec002de0 con 0x7f3bf8071370 2024-03-21T08:41:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.038+0000 7f3c14c00640 1 -- 172.21.15.28:0/2990122626 <== osd.3 v2:172.21.15.28:6816/2425819575 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3bec002de0 con 0x7f3bf8071370 2024-03-21T08:41:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7f3bf8071370 msgr2=0x7f3bf80737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 --> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5750073430 con 0x7f5730071370 2024-03-21T08:41:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7f3bf8071370 0x7f3bf80737b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bf8067bf0 msgr2=0x7f3bf806a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bf8067bf0 0x7f3bf806a0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3c08015630 tx=0x7f3c08015150 comp rx=0 tx=0).stop 2024-03-21T08:41:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5746000640 1 -- 172.21.15.28:0/3782254822 <== osd.6 v2:172.21.15.151:6808/3306491257 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5750073430 con 0x7f5730071370 2024-03-21T08:41:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 msgr2=0x7f3c181f3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c0e000640 1 --2- 172.21.15.28:0/2990122626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c1815df80 0x7f3c181f3150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c1001cd20 tx=0x7f3c10019220 comp rx=0 tx=0).stop 2024-03-21T08:41:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3c1dc00640 1 -- 172.21.15.28:0/2990122626 reap_dead start 2024-03-21T08:41:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5730071370 msgr2=0x7f57300737b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5730071370 0x7f57300737b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 --> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5560002de0 con 0x7f55680712a0 2024-03-21T08:41:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 --> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3f38152590 con 0x7f3f180712e0 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5730067bf0 msgr2=0x7f573006a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5730067bf0 0x7f573006a0b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f57480ae000 tx=0x7f5748004ac0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 msgr2=0x7f5750164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 --2- 172.21.15.28:0/3782254822 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5750142990 0x7f5750164340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e4e0 tx=0x7f573c00e9a0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f574e800640 1 -- 172.21.15.28:0/3782254822 reap_dead start 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 shutdown_connections 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 >> 172.21.15.28:0/3782254822 conn(0x7f57500674f0 msgr2=0x7f57501630c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 shutdown_connections 2024-03-21T08:41:29.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f5755200640 1 -- 172.21.15.28:0/3782254822 wait complete. 2024-03-21T08:41:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f557e000640 1 -- 172.21.15.28:0/4209491968 <== osd.4 v2:172.21.15.151:6809/3622853820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5560002de0 con 0x7f55680712a0 2024-03-21T08:41:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3f2ea00640 1 -- 172.21.15.28:0/3591342035 <== osd.1 v2:172.21.15.28:6800/4044131930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3f38152590 con 0x7f3f180712e0 2024-03-21T08:41:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f3f180712e0 msgr2=0x7f3f18073720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f3f180712e0 0x7f3f18073720 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7f55680712a0 msgr2=0x7f55680736e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7f55680712a0 0x7f55680736e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5568067ba0 msgr2=0x7f556806a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5568067ba0 0x7f556806a060 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5584050db0 tx=0x7f5584068e90 comp rx=0 tx=0).stop 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5580053c20 msgr2=0x7f55800db510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558b200640 1 --2- 172.21.15.28:0/4209491968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5580053c20 0x7f55800db510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f557800c420 tx=0x7f557800c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.042+0000 7f558a800640 1 -- 172.21.15.28:0/4209491968 reap_dead start 2024-03-21T08:41:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 shutdown_connections 2024-03-21T08:41:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 >> 172.21.15.28:0/4209491968 conn(0x7f55800559f0 msgr2=0x7f55800df7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 shutdown_connections 2024-03-21T08:41:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 >> 172.21.15.28:0/2990122626 conn(0x7f3c1806c8e0 msgr2=0x7f3c1815c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f18067b70 msgr2=0x7f3f1806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f18067b70 0x7f3f1806a030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3f380cc3f0 tx=0x7f3f28008040 comp rx=0 tx=0).stop 2024-03-21T08:41:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 msgr2=0x7f3f38155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 --2- 172.21.15.28:0/3591342035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f38071fc0 0x7f3f38155be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000cc80 tx=0x7f3f3000a430 comp rx=0 tx=0).stop 2024-03-21T08:41:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f37400640 1 -- 172.21.15.28:0/3591342035 reap_dead start 2024-03-21T08:41:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 shutdown_connections 2024-03-21T08:41:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 shutdown_connections 2024-03-21T08:41:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f558b200640 1 -- 172.21.15.28:0/4209491968 wait complete. 2024-03-21T08:41:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 shutdown_connections 2024-03-21T08:41:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 >> 172.21.15.28:0/3591342035 conn(0x7f3f380674f0 msgr2=0x7f3f38165b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3c0e000640 1 -- 172.21.15.28:0/2990122626 wait complete. 2024-03-21T08:41:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 shutdown_connections 2024-03-21T08:41:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.046+0000 7f3f3dc00640 1 -- 172.21.15.28:0/3591342035 wait complete. 2024-03-21T08:41:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.050+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 --> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f265815a080 con 0x7f26340712f0 2024-03-21T08:41:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.050+0000 7f264f400640 1 -- 172.21.15.28:0/2279723994 <== osd.7 v2:172.21.15.151:6824/1764908544 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f265815a080 con 0x7f26340712f0 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.050+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7f26340712f0 msgr2=0x7f2634073730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.050+0000 7f265de00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7f26340712f0 0x7f2634073730 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.054+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2634067b70 msgr2=0x7f263406a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.054+0000 7f265de00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2634067b70 0x7f263406a030 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f26400b7000 tx=0x7f26400b0000 comp rx=0 tx=0).stop 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.054+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 msgr2=0x7f265815f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.054+0000 7f265de00640 1 --2- 172.21.15.28:0/2279723994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2658140d40 0x7f265815f0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f264800e930 tx=0x7f264800edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.082 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:29.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-21T08:41:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.058+0000 7f2657400640 1 -- 172.21.15.28:0/2279723994 reap_dead start 2024-03-21T08:41:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.058+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 shutdown_connections 2024-03-21T08:41:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.058+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 >> 172.21.15.28:0/2279723994 conn(0x7f265806b770 msgr2=0x7f26581595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.058+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 shutdown_connections 2024-03-21T08:41:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.058+0000 7f265de00640 1 -- 172.21.15.28:0/2279723994 wait complete. 2024-03-21T08:41:29.084 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:29.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-21T08:41:29.085 INFO:teuthology.orchestra.run.smithi028.stdout:64424509449 2024-03-21T08:41:29.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-21T08:41:29.086 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:29.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-21T08:41:29.087 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:29.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-21T08:41:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:60129542153 2024-03-21T08:41:29.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-21T08:41:29.094 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:29.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-21T08:41:29.102 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:29.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-21T08:41:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 Processor -- start 2024-03-21T08:41:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 -- start start 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 0x7fbd880552f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd88057b40 con 0x7fbd88038520 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd88006250 con 0x7fbd880e76e0 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd95000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88057e20 con 0x7fbd880eb840 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd8e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd8d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd88038520 0x7fbd880eb130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fbd8d400640 1 -- 172.21.15.28:0/835391888 learned_addr learned my addr 172.21.15.28:0/835391888 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fdddba00640 1 Processor -- start 2024-03-21T08:41:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8e800640 1 -- 172.21.15.28:0/835391888 >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 msgr2=0x7fbd880552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8de00640 1 --2- 172.21.15.28:0/835391888 >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 0x7fbd880552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8ca00640 1 -- 172.21.15.28:0/835391888 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639281914 0 0) 0x7fbd88057b40 con 0x7fbd88038520 2024-03-21T08:41:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8e800640 1 --2- 172.21.15.28:0/835391888 >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 0x7fbd880552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8e800640 1 -- 172.21.15.28:0/835391888 >> v1:172.21.15.28:6789/0 conn(0x7fbd88038520 legacy=0x7fbd880eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8e800640 1 -- 172.21.15.28:0/835391888 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880064a0 con 0x7fbd880eb840 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fbd8de00640 1 --2- 172.21.15.28:0/835391888 >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 0x7fbd880552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.306+0000 7fdddba00640 1 -- start start 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fdddba00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fddd40b1fa0 0x7fddd416cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fdddba00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd406eec0 con 0x7fddd4165670 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fdddba00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd406f1a0 con 0x7fddd406e780 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fdddba00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd406f480 con 0x7fddd40b1fa0 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fddd40b1fa0 0x7fddd416cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fddd40b1fa0 0x7fddd416cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59146/0 (socket says 172.21.15.28:59146) 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 -- 172.21.15.28:0/215790005 learned_addr learned my addr 172.21.15.28:0/215790005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 -- 172.21.15.28:0/215790005 >> v1:172.21.15.151:6789/0 conn(0x7fddd406e780 legacy=0x7fddd40b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 -- 172.21.15.28:0/215790005 >> v1:172.21.15.28:6789/0 conn(0x7fddd4165670 legacy=0x7fddd40b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fddd9a00640 1 -- 172.21.15.28:0/215790005 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd406f760 con 0x7fddd40b1fa0 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 Processor -- start 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 -- start start 2024-03-21T08:41:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea4c050270 0x7fea4c0530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c0537e0 con 0x7fea4c0e2d80 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4c053ac0 con 0x7fea4c0df1a0 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.310+0000 7fea58e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4c054880 con 0x7fea4c050270 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea51e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fea4c0df1a0 0x7fea4c04fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39736/0 (socket says 172.21.15.28:39736) 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea51e00640 1 -- 172.21.15.28:0/3889495429 learned_addr learned my addr 172.21.15.28:0/3889495429 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 Processor -- start 2024-03-21T08:41:29.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea51400640 1 --2- 172.21.15.28:0/3889495429 >> v2:172.21.15.28:3300/0 conn(0x7fea4c050270 0x7fea4c0530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 -- start start 2024-03-21T08:41:29.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929693881 0 0) 0x7fea4c053ac0 con 0x7fea4c0df1a0 2024-03-21T08:41:29.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea38005740 con 0x7fea4c0df1a0 2024-03-21T08:41:29.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814366670 0 0) 0x7fea4c0537e0 con 0x7fea4c0e2d80 2024-03-21T08:41:29.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fbd8e800640 1 --2- 172.21.15.28:0/835391888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880069c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd800097c0 tx=0x7fbd8001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ab900ca4509e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea51400640 1 -- 172.21.15.28:0/3889495429 >> v1:172.21.15.151:6789/0 conn(0x7fea4c0df1a0 legacy=0x7fea4c04fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd7c071e70 0x7fcd7c159b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c0cc150 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c0cc430 con 0x7fcd7c0b0be0 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd81400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c06e930 con 0x7fcd7c071e70 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fea51400640 1 -- 172.21.15.28:0/3889495429 >> v1:172.21.15.28:6789/0 conn(0x7fea4c0e2d80 legacy=0x7fea4c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fbd8ca00640 1 -- 172.21.15.28:0/835391888 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd800b0070 con 0x7fbd880eb840 2024-03-21T08:41:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd7be00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fcd7c0b0be0 0x7fcd7c06e220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39738/0 (socket says 172.21.15.28:39738) 2024-03-21T08:41:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fbd8ca00640 1 -- 172.21.15.28:0/835391888 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd800ab600 con 0x7fbd880eb840 2024-03-21T08:41:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd8ca00640 1 -- 172.21.15.28:0/835391888 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd800ab9d0 con 0x7fbd880eb840 2024-03-21T08:41:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fea51400640 1 -- 172.21.15.28:0/3889495429 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c053da0 con 0x7fea4c050270 2024-03-21T08:41:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd7be00640 1 -- 172.21.15.28:0/2824001506 learned_addr learned my addr 172.21.15.28:0/2824001506 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.314+0000 7fcd80a00640 1 --2- 172.21.15.28:0/2824001506 >> v2:172.21.15.28:3300/0 conn(0x7fcd7c071e70 0x7fcd7c159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fcd7aa00640 1 -- 172.21.15.28:0/2824001506 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672671583 0 0) 0x7fcd7c0cc430 con 0x7fcd7c0b0be0 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fcd7aa00640 1 -- 172.21.15.28:0/2824001506 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd60003530 con 0x7fcd7c0b0be0 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fcd80a00640 1 -- 172.21.15.28:0/2824001506 >> v1:172.21.15.151:6789/0 conn(0x7fcd7c0b0be0 legacy=0x7fcd7c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fcd80a00640 1 -- 172.21.15.28:0/2824001506 >> v1:172.21.15.28:6789/0 conn(0x7fcd7c0b1fc0 legacy=0x7fcd7c071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fcd80a00640 1 -- 172.21.15.28:0/2824001506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c0cc710 con 0x7fcd7c071e70 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fddd9a00640 1 --2- 172.21.15.28:0/215790005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd416cc10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009870 tx=0x7fddc001c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26caaa456bb773ae server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fddd3400640 1 -- 172.21.15.28:0/215790005 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc00b0070 con 0x7fddd40b1fa0 2024-03-21T08:41:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 -- 172.21.15.28:0/835391888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 msgr2=0x7fbd880069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 --2- 172.21.15.28:0/835391888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880069c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd800097c0 tx=0x7fbd8001c410 comp rx=0 tx=0).stop 2024-03-21T08:41:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 -- 172.21.15.28:0/835391888 shutdown_connections 2024-03-21T08:41:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 --2- 172.21.15.28:0/835391888 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880069c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 --2- 172.21.15.28:0/835391888 >> v2:172.21.15.151:3300/0 conn(0x7fbd880e76e0 0x7fbd880552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 -- 172.21.15.28:0/835391888 >> 172.21.15.28:0/835391888 conn(0x7fbd8803aa40 msgr2=0x7fbd8803ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 -- 172.21.15.28:0/835391888 shutdown_connections 2024-03-21T08:41:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fbd95000640 1 -- 172.21.15.28:0/835391888 wait complete. 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fddd3400640 1 -- 172.21.15.28:0/215790005 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc00ab600 con 0x7fddd40b1fa0 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.318+0000 7fddd3400640 1 -- 172.21.15.28:0/215790005 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc00b0070 con 0x7fddd40b1fa0 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 -- 172.21.15.28:0/215790005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 msgr2=0x7fddd416cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 --2- 172.21.15.28:0/215790005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd416cc10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009870 tx=0x7fddc001c4b0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 -- 172.21.15.28:0/215790005 shutdown_connections 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 --2- 172.21.15.28:0/215790005 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd416cc10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea51400640 1 --2- 172.21.15.28:0/3889495429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c050270 0x7fea4c0530e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea54067f00 tx=0x7fea540976b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777d3bc09fc889f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 -- 172.21.15.28:0/215790005 >> 172.21.15.28:0/215790005 conn(0x7fddd406b170 msgr2=0x7fddd406b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 -- 172.21.15.28:0/215790005 shutdown_connections 2024-03-21T08:41:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5406f480 con 0x7fea4c050270 2024-03-21T08:41:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5406f620 con 0x7fea4c050270 2024-03-21T08:41:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fbd95000640 1 Processor -- start 2024-03-21T08:41:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea50a00640 1 -- 172.21.15.28:0/3889495429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5406f9d0 con 0x7fea4c050270 2024-03-21T08:41:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fbd95000640 1 -- start start 2024-03-21T08:41:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea58e00640 1 -- 172.21.15.28:0/3889495429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c050270 msgr2=0x7fea4c0530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea58e00640 1 --2- 172.21.15.28:0/3889495429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c050270 0x7fea4c0530e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea54067f00 tx=0x7fea540976b0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fdddba00640 1 -- 172.21.15.28:0/215790005 wait complete. 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd80a00640 1 --2- 172.21.15.28:0/2824001506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c071e70 0x7fcd7c159b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0097c0 tx=0x7fcd6c024cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb01c831f7f9480f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd7aa00640 1 -- 172.21.15.28:0/2824001506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0ba070 con 0x7fcd7c071e70 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd7aa00640 1 -- 172.21.15.28:0/2824001506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c0ada70 con 0x7fcd7c071e70 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd7aa00640 1 -- 172.21.15.28:0/2824001506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0ba070 con 0x7fcd7c071e70 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd81400640 1 -- 172.21.15.28:0/2824001506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c071e70 msgr2=0x7fcd7c159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd81400640 1 --2- 172.21.15.28:0/2824001506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c071e70 0x7fcd7c159b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0097c0 tx=0x7fcd6c024cd0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd81400640 1 -- 172.21.15.28:0/2824001506 shutdown_connections 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd81400640 1 --2- 172.21.15.28:0/2824001506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c071e70 0x7fcd7c159b70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fcd81400640 1 -- 172.21.15.28:0/2824001506 >> 172.21.15.28:0/2824001506 conn(0x7fcd7c0b14a0 msgr2=0x7fcd7c0b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea58e00640 1 -- 172.21.15.28:0/3889495429 shutdown_connections 2024-03-21T08:41:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea58e00640 1 --2- 172.21.15.28:0/3889495429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c050270 0x7fea4c0530e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.322+0000 7fea58e00640 1 -- 172.21.15.28:0/3889495429 >> 172.21.15.28:0/3889495429 conn(0x7fea4c054b00 msgr2=0x7fea4c054f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fea58e00640 1 -- 172.21.15.28:0/3889495429 shutdown_connections 2024-03-21T08:41:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fea58e00640 1 -- 172.21.15.28:0/3889495429 wait complete. 2024-03-21T08:41:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 0x7fbd880dc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 Processor -- start 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880e5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88007ae0 con 0x7fbd880e76e0 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88007960 con 0x7fbd88038520 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd95000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880f44d0 con 0x7fbd880eb840 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 0x7fbd880dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51612/0 (socket says 172.21.15.28:51612) 2024-03-21T08:41:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8d400640 1 -- 172.21.15.28:0/2962880123 learned_addr learned my addr 172.21.15.28:0/2962880123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 -- start start 2024-03-21T08:41:29.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 -- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 msgr2=0x7fbd880e5680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880e5680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 -- 172.21.15.28:0/2962880123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 msgr2=0x7fbd8804ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 -- 172.21.15.28:0/2962880123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8803bc10 con 0x7fbd88038520 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8d400640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fcd81400640 1 -- 172.21.15.28:0/2824001506 shutdown_connections 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 0x7fddd415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd4159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 0x7fddd415e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd416fd00 con 0x7fddd4165670 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd416fb80 con 0x7fddd40b1fa0 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fdddba00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd416fe80 con 0x7fddd406e780 2024-03-21T08:41:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fcd81400640 1 -- 172.21.15.28:0/2824001506 wait complete. 2024-03-21T08:41:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 0x7fddd415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 0x7fddd415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 0x7fddd415e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51614/0 (socket says 172.21.15.28:51614) 2024-03-21T08:41:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9a00640 1 -- 172.21.15.28:0/1548036992 learned_addr learned my addr 172.21.15.28:0/1548036992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fbd8de00640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 0x7fbd880dc560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd90050d50 tx=0x7fbd90068cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 -- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 msgr2=0x7fddd4159200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd4159200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 -- 172.21.15.28:0/1548036992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 msgr2=0x7fddd415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 0x7fddd415e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 -- 172.21.15.28:0/1548036992 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd415a5d0 con 0x7fddd406e780 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fddd9000640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 0x7fddd415b650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00be20 tx=0x7fddcc009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90073c70 con 0x7fbd88038520 2024-03-21T08:41:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd90073e30 con 0x7fbd88038520 2024-03-21T08:41:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc019070 con 0x7fddd406e780 2024-03-21T08:41:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc014440 con 0x7fddd406e780 2024-03-21T08:41:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc014870 con 0x7fddd406e780 2024-03-21T08:41:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90074220 con 0x7fbd88038520 2024-03-21T08:41:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd415a8b0 con 0x7fddd406e780 2024-03-21T08:41:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd415ae70 con 0x7fddd406e780 2024-03-21T08:41:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.326+0000 7fea58e00640 1 Processor -- start 2024-03-21T08:41:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 -- start start 2024-03-21T08:41:29.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fcd81400640 1 Processor -- start 2024-03-21T08:41:29.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fcd81400640 1 -- start start 2024-03-21T08:41:29.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88052810 con 0x7fbd88038520 2024-03-21T08:41:29.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea4c050270 0x7fea4c04cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c0e2d80 0x7fea4c0d42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c006f70 con 0x7fea4c0e2d80 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c006df0 con 0x7fea4c050270 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea58e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c0ed130 con 0x7fea4c0df1a0 2024-03-21T08:41:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88052dd0 con 0x7fbd88038520 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37322/0 (socket says 172.21.15.28:37322) 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 -- 172.21.15.28:0/2779795914 learned_addr learned my addr 172.21.15.28:0/2779795914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea52800640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c0e2d80 0x7fea4c0d42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 -- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea4c050270 msgr2=0x7fea4c04cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51e00640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea4c050270 0x7fea4c04cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd880e6ea0 con 0x7fbd88038520 2024-03-21T08:41:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea4c050270 0x7fea4c04cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 -- 172.21.15.28:0/2779795914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c0e2d80 msgr2=0x7fea4c0d42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c0e2d80 0x7fea4c0d42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 -- 172.21.15.28:0/2779795914 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c04d370 con 0x7fea4c0df1a0 2024-03-21T08:41:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fea51400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea5402d610 tx=0x7fea54094470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fcd81400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd7c071e70 0x7fcd7c148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fcd81400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 0x7fcd7c134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.330+0000 7fcd81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 0x7fcd7c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd81400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0cdd90 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd81400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0cdc10 con 0x7fcd7c0b0be0 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd7b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 0x7fcd7c134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbd90093440 con 0x7fbd88038520 2024-03-21T08:41:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd7b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 0x7fcd7c134570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59174/0 (socket says 172.21.15.28:59174) 2024-03-21T08:41:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd7b400640 1 -- 172.21.15.28:0/663955604 learned_addr learned my addr 172.21.15.28:0/663955604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd7be00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd7c071e70 0x7fcd7c148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd404f410 con 0x7fddd406e780 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0cdf10 con 0x7fcd7c071e70 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 0x7fcd7c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 -- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd7c071e70 msgr2=0x7fcd7c148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd7c071e70 0x7fcd7c148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 -- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 msgr2=0x7fcd7c134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 0x7fcd7c134570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c149990 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fddcc01c070 con 0x7fddd406e780 2024-03-21T08:41:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd80a00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 0x7fcd7c157d30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0cc360 tx=0x7fcd6c021a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fbd86000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 0x7fbd7006a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea54135020 con 0x7fea4c0df1a0 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0c2020 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbd90101610 con 0x7fbd88038520 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fbd8d400640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 0x7fbd7006a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fddd0c00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 0x7fddb406a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fddcc086170 con 0x7fddd406e780 2024-03-21T08:41:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fddd3e00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 0x7fddb406a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c004340 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0ba040 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c0b42e0 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c0b48a0 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c04f410 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.334+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea54131670 con 0x7fea4c0df1a0 2024-03-21T08:41:29.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea54131a20 con 0x7fea4c0df1a0 2024-03-21T08:41:29.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fea58e00640 1 -- 172.21.15.28:0/2779795914 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c04d650 con 0x7fea4c0df1a0 2024-03-21T08:41:29.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fea58e00640 1 -- 172.21.15.28:0/2779795914 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c04dc10 con 0x7fea4c0df1a0 2024-03-21T08:41:29.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fbd8d400640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 0x7fbd7006a0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd88183f80 tx=0x7fbd7c007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fddd3e00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 0x7fddb406a0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000f70 tx=0x7fddc40079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fea58e00640 1 -- 172.21.15.28:0/2779795914 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c0053d0 con 0x7fea4c0df1a0 2024-03-21T08:41:29.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcd6c0c5020 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7f276cc00640 1 Processor -- start 2024-03-21T08:41:29.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.338+0000 7f276cc00640 1 -- start start 2024-03-21T08:41:29.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7fcd73e00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 0x7fcd5c06a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd6c128750 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f276815bd20 0x7f27680b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f27680b0e80 0x7f27680ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27680cee30 con 0x7f276815bd20 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276815fba0 con 0x7f27680b0e80 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f276cc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276815fe80 con 0x7f27680cea00 2024-03-21T08:41:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7fcd7be00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 0x7fcd5c06a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2765400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f27680b0e80 0x7f27680ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37342/0 (socket says 172.21.15.28:37342) 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 -- 172.21.15.28:0/1838033916 learned_addr learned my addr 172.21.15.28:0/1838033916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2765e00640 1 --2- 172.21.15.28:0/1838033916 >> v2:172.21.15.28:3300/0 conn(0x7f276815bd20 0x7f27680b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 -- 172.21.15.28:0/1838033916 >> v2:172.21.15.151:3300/0 conn(0x7f27680b0e80 msgr2=0x7f27680ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 --2- 172.21.15.28:0/1838033916 >> v2:172.21.15.151:3300/0 conn(0x7f27680b0e80 0x7f27680ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 -- 172.21.15.28:0/1838033916 >> v2:172.21.15.28:3300/0 conn(0x7f276815bd20 msgr2=0x7f27680b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 --2- 172.21.15.28:0/1838033916 >> v2:172.21.15.28:3300/0 conn(0x7f276815bd20 0x7f27680b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7f2766800640 1 -- 172.21.15.28:0/1838033916 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2768160160 con 0x7f27680cea00 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.342+0000 7fcd7be00640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 0x7fcd5c06a030 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcd74001570 tx=0x7fcd74001aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.346+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbd900d1c30 con 0x7fbd88038520 2024-03-21T08:41:29.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.346+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fddcc056260 con 0x7fddd406e780 2024-03-21T08:41:29.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.346+0000 7f2766800640 1 --2- 172.21.15.28:0/1838033916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f276806fb50 tx=0x7f276001c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726a4c6077d13300 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.346+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fea54138020 con 0x7fea4c0df1a0 2024-03-21T08:41:29.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.346+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcd6c0cc070 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f2764a00640 1 -- 172.21.15.28:0/1838033916 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27600ac070 con 0x7f27680cea00 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f2764a00640 1 -- 172.21.15.28:0/1838033916 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27600a8420 con 0x7f27680cea00 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f2764a00640 1 -- 172.21.15.28:0/1838033916 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27600a87f0 con 0x7f27680cea00 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7fea4a000640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 0x7fea3406a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fea5419bc70 con 0x7fea4c0df1a0 2024-03-21T08:41:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fea5413f030 con 0x7fea4c0df1a0 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 -- 172.21.15.28:0/1838033916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 msgr2=0x7f27680b19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 --2- 172.21.15.28:0/1838033916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f276806fb50 tx=0x7f276001c730 comp rx=0 tx=0).stop 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 -- 172.21.15.28:0/1838033916 shutdown_connections 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 --2- 172.21.15.28:0/1838033916 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680b19e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 --2- 172.21.15.28:0/1838033916 >> v2:172.21.15.151:3300/0 conn(0x7f27680b0e80 0x7f27680ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 --2- 172.21.15.28:0/1838033916 >> v2:172.21.15.28:3300/0 conn(0x7f276815bd20 0x7f27680b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 -- 172.21.15.28:0/1838033916 >> 172.21.15.28:0/1838033916 conn(0x7f27680d1ab0 msgr2=0x7f27680d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.350+0000 7f276cc00640 1 -- 172.21.15.28:0/1838033916 shutdown_connections 2024-03-21T08:41:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.354+0000 7f276cc00640 1 -- 172.21.15.28:0/1838033916 wait complete. 2024-03-21T08:41:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.354+0000 7fea51e00640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 0x7fea3406a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.354+0000 7f276cc00640 1 Processor -- start 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.354+0000 7f276cc00640 1 -- start start 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 0x7f2768154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276815c280 con 0x7f27680b0e80 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276815c100 con 0x7f276815bd20 2024-03-21T08:41:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f276cc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276808afc0 con 0x7f27680cea00 2024-03-21T08:41:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7fea51e00640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 0x7fea3406a030 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea40004980 tx=0x7fea40005c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f2765400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f2766800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f2766800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59192/0 (socket says 172.21.15.28:59192) 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.358+0000 7f2766800640 1 -- 172.21.15.28:0/4094616351 learned_addr learned my addr 172.21.15.28:0/4094616351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2765e00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 0x7f2768154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 -- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 msgr2=0x7f27680cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 -- 172.21.15.28:0/4094616351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 msgr2=0x7f2768154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 0x7f2768154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 -- 172.21.15.28:0/4094616351 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27681fa550 con 0x7f276815bd20 2024-03-21T08:41:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2765e00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 0x7f2768154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f2766800640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f276001ade0 tx=0x7f27600194d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27600ac040 con 0x7f276815bd20 2024-03-21T08:41:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27600bd450 con 0x7f276815bd20 2024-03-21T08:41:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27600bd840 con 0x7f276815bd20 2024-03-21T08:41:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27680ca020 con 0x7f276815bd20 2024-03-21T08:41:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.362+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27680ca5e0 con 0x7f276815bd20 2024-03-21T08:41:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.366+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f27600af070 con 0x7f276815bd20 2024-03-21T08:41:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.366+0000 7f275e000640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 0x7f2744069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.366+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2760120710 con 0x7f276815bd20 2024-03-21T08:41:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.366+0000 7f2765e00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 0x7f2744069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.370+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276804f4f0 con 0x7f276815bd20 2024-03-21T08:41:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.370+0000 7f2765e00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 0x7f2744069fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2758009eb0 tx=0x7f2758009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.374+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f27600c4030 con 0x7f276815bd20 2024-03-21T08:41:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 Processor -- start 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 -- start start 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffa2c15bd20 0x7ffa2c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ffa2c0b0e80 0x7ffa2c0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0cede0 con 0x7ffa2c0cea00 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2c08bfd0 con 0x7ffa2c15bd20 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa31c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2c15fba0 con 0x7ffa2c0b0e80 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa2b400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ffa2c0cea00 0x7ffa2c0b19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39750/0 (socket says 172.21.15.28:39750) 2024-03-21T08:41:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7ffa2b400640 1 -- 172.21.15.28:0/1561734892 learned_addr learned my addr 172.21.15.28:0/1561734892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.394+0000 7fd490a00640 1 Processor -- start 2024-03-21T08:41:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd490a00640 1 -- start start 2024-03-21T08:41:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2a000640 1 --2- 172.21.15.28:0/1561734892 >> v2:172.21.15.151:3300/0 conn(0x7ffa2c0b0e80 0x7ffa2c0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614428870 0 0) 0x7ffa2c0cede0 con 0x7ffa2c0cea00 2024-03-21T08:41:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa10003530 con 0x7ffa2c0cea00 2024-03-21T08:41:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> v2:172.21.15.28:3300/0 conn(0x7ffa2c15bd20 0x7ffa2c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v1:172.21.15.151:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3510342801 0 0) 0x7ffa10003530 con 0x7ffa2c0cea00 2024-03-21T08:41:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 >> v2:172.21.15.151:3300/0 conn(0x7ffa2c0b0e80 msgr2=0x7ffa2c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 --2- 172.21.15.28:0/1561734892 >> v2:172.21.15.151:3300/0 conn(0x7ffa2c0b0e80 0x7ffa2c0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 >> v2:172.21.15.28:3300/0 conn(0x7ffa2c15bd20 msgr2=0x7ffa2c0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd490a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd48c08a570 0x7fd48c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd490a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0af560 con 0x7fd48c172960 2024-03-21T08:41:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd490a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c06f540 con 0x7fd48c172250 2024-03-21T08:41:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd490a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48c06e6c0 con 0x7fd48c08a570 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 --2- 172.21.15.28:0/1561734892 >> v2:172.21.15.28:3300/0 conn(0x7ffa2c15bd20 0x7ffa2c0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 --> v1:172.21.15.151:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c15fe80 con 0x7ffa2c0cea00 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v1:172.21.15.151:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7ffa24001680 con 0x7ffa2c0cea00 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 >> v1:172.21.15.151:6789/0 conn(0x7ffa2c0cea00 legacy=0x7ffa2c0b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 0x7ffa10003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10004810 0x7ffa10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100051d0 0x7ffa1000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15fba0 con 0x7ffa10004810 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1000c020 con 0x7ffa100039c0 2024-03-21T08:41:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1000c220 con 0x7ffa100051d0 2024-03-21T08:41:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> v2:172.21.15.28:3300/0 conn(0x7ffa2c15bd20 0x7ffa2c0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd489e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd48c172250 0x7fd48c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39766/0 (socket says 172.21.15.28:39766) 2024-03-21T08:41:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd489e00640 1 -- 172.21.15.28:0/1263432832 learned_addr learned my addr 172.21.15.28:0/1263432832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7fd489400640 1 --2- 172.21.15.28:0/1263432832 >> v2:172.21.15.28:3300/0 conn(0x7fd48c08a570 0x7fd48c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 0x7ffa10003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.402+0000 7fd489400640 1 -- 172.21.15.28:0/1263432832 >> v1:172.21.15.151:6789/0 conn(0x7fd48c172250 legacy=0x7fd48c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.402+0000 7fd489400640 1 -- 172.21.15.28:0/1263432832 >> v1:172.21.15.28:6789/0 conn(0x7fd48c172960 legacy=0x7fd48c0b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.402+0000 7fd489400640 1 -- 172.21.15.28:0/1263432832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c06fd40 con 0x7fd48c08a570 2024-03-21T08:41:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2b400640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100051d0 0x7ffa1000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 -- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100051d0 msgr2=0x7ffa1000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100051d0 0x7ffa1000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 -- 172.21.15.28:0/1561734892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10004810 msgr2=0x7ffa10004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa10004810 0x7ffa10004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.398+0000 7ffa2aa00640 1 -- 172.21.15.28:0/1561734892 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c15fe80 con 0x7ffa100039c0 2024-03-21T08:41:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa2b400640 1 -- 172.21.15.28:0/1561734892 reap_dead start 2024-03-21T08:41:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 0x7ffa10003de0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c490 tx=0x7ffa1c01d120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff1b0af1146e961f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0b1070 con 0x7ffa100039c0 2024-03-21T08:41:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c0ac600 con 0x7ffa100039c0 2024-03-21T08:41:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7fd489400640 1 --2- 172.21.15.28:0/1263432832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c0af020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd48c06e840 tx=0x7fd47801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d1fe0597b2013d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 -- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 msgr2=0x7ffa10003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa29600640 1 -- 172.21.15.28:0/1561734892 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0b1070 con 0x7ffa100039c0 2024-03-21T08:41:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 0x7ffa10003de0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c490 tx=0x7ffa1c01d120 comp rx=0 tx=0).stop 2024-03-21T08:41:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7fd488a00640 1 -- 172.21.15.28:0/1263432832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780b0070 con 0x7fd48c08a570 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 -- 172.21.15.28:0/1561734892 shutdown_connections 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 --2- 172.21.15.28:0/1561734892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa100039c0 0x7ffa10003de0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 -- 172.21.15.28:0/1561734892 >> 172.21.15.28:0/1561734892 conn(0x7ffa2c0d1ab0 msgr2=0x7ffa2c0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 -- 172.21.15.28:0/1561734892 shutdown_connections 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.406+0000 7ffa31c00640 1 -- 172.21.15.28:0/1561734892 wait complete. 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd488a00640 1 -- 172.21.15.28:0/1263432832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780ab600 con 0x7fd48c08a570 2024-03-21T08:41:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd488a00640 1 -- 172.21.15.28:0/1263432832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780ab9d0 con 0x7fd48c08a570 2024-03-21T08:41:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 -- 172.21.15.28:0/1263432832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 msgr2=0x7fd48c0af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 --2- 172.21.15.28:0/1263432832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c0af020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd48c06e840 tx=0x7fd47801c350 comp rx=0 tx=0).stop 2024-03-21T08:41:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 -- 172.21.15.28:0/1263432832 shutdown_connections 2024-03-21T08:41:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 --2- 172.21.15.28:0/1263432832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c0af020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 -- 172.21.15.28:0/1263432832 >> 172.21.15.28:0/1263432832 conn(0x7fd48c06b770 msgr2=0x7fd48c0b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 -- 172.21.15.28:0/1263432832 shutdown_connections 2024-03-21T08:41:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7fd490a00640 1 -- 172.21.15.28:0/1263432832 wait complete. 2024-03-21T08:41:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7ffa31c00640 1 Processor -- start 2024-03-21T08:41:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.410+0000 7ffa31c00640 1 -- start start 2024-03-21T08:41:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa31c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 0x7ffa2c0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa31c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 0x7ffa2c15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 0x7ffa2c0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 0x7ffa2c0b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37364/0 (socket says 172.21.15.28:37364) 2024-03-21T08:41:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7fd490a00640 1 Processor -- start 2024-03-21T08:41:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7fd490a00640 1 -- start start 2024-03-21T08:41:29.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa2aa00640 1 -- 172.21.15.28:0/2124986392 learned_addr learned my addr 172.21.15.28:0/2124986392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.414+0000 7ffa2a000640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 0x7ffa2c15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c0cb1e0 0x7ffa2c09ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cc520 con 0x7ffa2c0cb1e0 2024-03-21T08:41:29.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c160440 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cc6a0 con 0x7ffa100039c0 2024-03-21T08:41:29.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 0x7fd48c16a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 0x7fd48c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c06e910 con 0x7fd48c08a570 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c070300 con 0x7fd48c172960 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd490a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c08a9d0 con 0x7fd48c172250 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51692/0 (socket says 172.21.15.28:51692) 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 -- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 msgr2=0x7ffa2c0b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 0x7ffa2c0b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 -- 172.21.15.28:0/2124986392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c0cb1e0 msgr2=0x7ffa2c09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c0cb1e0 0x7ffa2c09ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0ca140 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 -- 172.21.15.28:0/3926303028 learned_addr learned my addr 172.21.15.28:0/3926303028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa2a000640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 0x7ffa2c15a6f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e750 tx=0x7ffa1800ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489400640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 0x7fd48c16a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd48a800640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 0x7fd48c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18017070 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800ee60 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800cb80 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 -- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 msgr2=0x7fd48c16a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 0x7fd48c16a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 -- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 msgr2=0x7fd48c15ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 0x7fd48c15ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489e00640 1 -- 172.21.15.28:0/3926303028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c1f8de0 con 0x7fd48c08a570 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd489400640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 0x7fd48c16a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7fd48a800640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 0x7fd48c15ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c0ca420 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.418+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c0ca9e0 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.422+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c04f4f0 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.422+0000 7fd489e00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00c530 tx=0x7fd47c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.422+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ffa1801a070 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c017070 con 0x7fd48c08a570 2024-03-21T08:41:29.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c005cf0 con 0x7fd48c08a570 2024-03-21T08:41:29.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7ffa22a00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 0x7ffa0c06a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c007270 con 0x7fd48c08a570 2024-03-21T08:41:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c1f90c0 con 0x7fd48c08a570 2024-03-21T08:41:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 0x7ffa0c06a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ffa18086240 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7ffa2aa00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 0x7ffa0c06a0b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0011a0 tx=0x7ffa1c004a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c1f9680 con 0x7fd48c08a570 2024-03-21T08:41:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.426+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c073430 con 0x7fd48c08a570 2024-03-21T08:41:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.430+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffa18056330 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd47c01a070 con 0x7fd48c08a570 2024-03-21T08:41:29.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd482000640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 0x7fd46c06a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd47c007070 con 0x7fd48c08a570 2024-03-21T08:41:29.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd489400640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 0x7fd46c06a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd489400640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 0x7fd46c06a1e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4780ad000 tx=0x7fd4780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.438+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd47c055890 con 0x7fd48c08a570 2024-03-21T08:41:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.454+0000 7f871e200640 1 Processor -- start 2024-03-21T08:41:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.454+0000 7f871e200640 1 -- start start 2024-03-21T08:41:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f871e200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f871815e3a0 0x7f87180b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f871e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8717400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f87180ced70 0x7f87180d1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39778/0 (socket says 172.21.15.28:39778) 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8716a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f871815e3a0 0x7f87180b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8716a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f871815e3a0 0x7f87180b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59246/0 (socket says 172.21.15.28:59246) 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8717e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8717e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37390/0 (socket says 172.21.15.28:37390) 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8717400640 1 -- 172.21.15.28:0/1868546851 learned_addr learned my addr 172.21.15.28:0/1868546851 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8718161a20 con 0x7f87180ced70 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87180d1600 con 0x7f871815e3a0 2024-03-21T08:41:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b3730 con 0x7f87180d0c50 2024-03-21T08:41:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8716000640 1 -- 172.21.15.28:0/1868546851 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037236303 0 0) 0x7f8718161a20 con 0x7f87180ced70 2024-03-21T08:41:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8716000640 1 -- 172.21.15.28:0/1868546851 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc003530 con 0x7f87180ced70 2024-03-21T08:41:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.458+0000 7f8716a00640 1 -- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 msgr2=0x7f87180d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.462+0000 7f8716a00640 1 --2- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180d10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.462+0000 7f8716a00640 1 -- 172.21.15.28:0/1868546851 >> v1:172.21.15.151:6789/0 conn(0x7f87180ced70 legacy=0x7f87180d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.462+0000 7f8716a00640 1 -- 172.21.15.28:0/1868546851 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180d2170 con 0x7f871815e3a0 2024-03-21T08:41:29.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f8716a00640 1 --2- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f871815e3a0 0x7f87180b31f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f87080097c0 tx=0x7f870801c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92785aabed329b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f8716000640 1 -- 172.21.15.28:0/1868546851 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080a7070 con 0x7f871815e3a0 2024-03-21T08:41:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f8716000640 1 -- 172.21.15.28:0/1868546851 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87080b0070 con 0x7f871815e3a0 2024-03-21T08:41:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f8716000640 1 -- 172.21.15.28:0/1868546851 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080ab890 con 0x7f871815e3a0 2024-03-21T08:41:29.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f871815e3a0 msgr2=0x7f87180b31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 --2- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f871815e3a0 0x7f87180b31f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f87080097c0 tx=0x7f870801c490 comp rx=0 tx=0).stop 2024-03-21T08:41:29.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 shutdown_connections 2024-03-21T08:41:29.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 --2- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180d10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 --2- 172.21.15.28:0/1868546851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f871815e3a0 0x7f87180b31f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 >> 172.21.15.28:0/1868546851 conn(0x7f8718072aa0 msgr2=0x7f8718072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 shutdown_connections 2024-03-21T08:41:29.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.466+0000 7f871e200640 1 -- 172.21.15.28:0/1868546851 wait complete. 2024-03-21T08:41:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.470+0000 7f871e200640 1 Processor -- start 2024-03-21T08:41:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.470+0000 7f871e200640 1 -- start start 2024-03-21T08:41:29.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.470+0000 7f871e200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 0x7f87181570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.470+0000 7f871e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.470+0000 7f871e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 0x7f87180b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 0x7f87181570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 0x7f87181570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:59260/0 (socket says 172.21.15.28:59260) 2024-03-21T08:41:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717400640 1 -- 172.21.15.28:0/1831240511 learned_addr learned my addr 172.21.15.28:0/1831240511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:29.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180cf150 con 0x7f871815e3a0 2024-03-21T08:41:29.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871815f280 con 0x7f87180ced70 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180cf2d0 con 0x7f87180d0c50 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8716a00640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 0x7f87180b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 -- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 msgr2=0x7f87180b8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180b8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 -- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 msgr2=0x7f87181570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 0x7f87181570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718156030 con 0x7f871815e3a0 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f8717e00640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 0x7f87180b5ad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f871000efc0 tx=0x7f871000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.474+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710024070 con 0x7f871815e3a0 2024-03-21T08:41:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.478+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8718156310 con 0x7f871815e3a0 2024-03-21T08:41:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.478+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87181568d0 con 0x7f871815e3a0 2024-03-21T08:41:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.478+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871001f440 con 0x7f871815e3a0 2024-03-21T08:41:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.478+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710024070 con 0x7f871815e3a0 2024-03-21T08:41:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.482+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f00052a0 con 0x7f871815e3a0 2024-03-21T08:41:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.482+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f87100143e0 con 0x7f871815e3a0 2024-03-21T08:41:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.482+0000 7f870f400640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 0x7f86f806a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.482+0000 7f8717400640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 0x7f86f806a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.482+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f871000cd30 con 0x7f871815e3a0 2024-03-21T08:41:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.486+0000 7f8717400640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 0x7f86f806a410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8704007fd0 tx=0x7f8704009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.494+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8710010020 con 0x7f871815e3a0 2024-03-21T08:41:29.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.681+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fbd88038250 con 0x7fbd88038520 2024-03-21T08:41:29.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.681+0000 7fbd86000640 1 -- 172.21.15.28:0/2962880123 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd900d1530 con 0x7fbd88038520 2024-03-21T08:41:29.685 INFO:teuthology.orchestra.run.smithi028.stdout:64424509449 2024-03-21T08:41:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 msgr2=0x7fbd7006a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 0x7fbd7006a0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd88183f80 tx=0x7fbd7c007ee0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 msgr2=0x7fbd880dc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 0x7fbd880dc560 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd90050d50 tx=0x7fbd90068cb0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 shutdown_connections 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd70067c20 0x7fbd7006a0e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd880eb840 0x7fbd880e5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd880e76e0 0x7fbd8804ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 --2- 172.21.15.28:0/2962880123 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd88038520 0x7fbd880dc560 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 >> 172.21.15.28:0/2962880123 conn(0x7fbd8803aa40 msgr2=0x7fbd880e17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 shutdown_connections 2024-03-21T08:41:29.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.685+0000 7fbd95000640 1 -- 172.21.15.28:0/2962880123 wait complete. 2024-03-21T08:41:29.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.693+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fddd40b1a30 con 0x7fddd406e780 2024-03-21T08:41:29.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.693+0000 7fddd0c00640 1 -- 172.21.15.28:0/1548036992 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddcc05a300 con 0x7fddd406e780 2024-03-21T08:41:29.699 INFO:teuthology.orchestra.run.smithi028.stdout:60129542152 2024-03-21T08:41:29.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcd7c0b1920 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fcd73e00640 1 -- 172.21.15.28:0/663955604 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd6c01e8c0 con 0x7fcd7c0b1fc0 2024-03-21T08:41:29.702 INFO:teuthology.orchestra.run.smithi028.stdout:60129542152 2024-03-21T08:41:29.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 msgr2=0x7fddb406a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 0x7fddb406a0b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000f70 tx=0x7fddc40079c0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 msgr2=0x7fddd415b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 0x7fddd415b650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00be20 tx=0x7fddcc009f70 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 shutdown_connections 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddd4165670 0x7fddd415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fddd40b1fa0 0x7fddd4159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fddb4067bf0 0x7fddb406a0b0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000f70 tx=0x7fddc40079c0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 --2- 172.21.15.28:0/1548036992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fddd406e780 0x7fddd415b650 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00be20 tx=0x7fddcc009f70 comp rx=0 tx=0).stop 2024-03-21T08:41:29.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.697+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 >> 172.21.15.28:0/1548036992 conn(0x7fddd406b170 msgr2=0x7fddd4164200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 shutdown_connections 2024-03-21T08:41:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fdddba00640 1 -- 172.21.15.28:0/1548036992 wait complete. 2024-03-21T08:41:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 msgr2=0x7fcd5c06a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 0x7fcd5c06a030 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcd74001570 tx=0x7fcd74001aa0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 msgr2=0x7fcd7c157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 0x7fcd7c157d30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0cc360 tx=0x7fcd6c021a70 comp rx=0 tx=0).stop 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 shutdown_connections 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd7c0b1fc0 0x7fcd7c157d30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd7c0b0be0 0x7fcd7c134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd5c067b70 0x7fcd5c06a030 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 --2- 172.21.15.28:0/663955604 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd7c071e70 0x7fcd7c148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 >> 172.21.15.28:0/663955604 conn(0x7fcd7c0b14a0 msgr2=0x7fcd7c153bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 shutdown_connections 2024-03-21T08:41:29.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fcd81400640 1 -- 172.21.15.28:0/663955604 wait complete. 2024-03-21T08:41:29.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fea58e00640 1 -- 172.21.15.28:0/2779795914 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fea4c052c90 con 0x7fea4c0df1a0 2024-03-21T08:41:29.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.701+0000 7fea4a000640 1 -- 172.21.15.28:0/2779795914 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea54172070 con 0x7fea4c0df1a0 2024-03-21T08:41:29.706 INFO:teuthology.orchestra.run.smithi028.stdout:64424509448 2024-03-21T08:41:29.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 msgr2=0x7fea3406a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 0x7fea3406a030 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea40004980 tx=0x7fea40005c00 comp rx=0 tx=0).stop 2024-03-21T08:41:29.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 msgr2=0x7fea4c0d3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea5402d610 tx=0x7fea54094470 comp rx=0 tx=0).stop 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 shutdown_connections 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea4c0e2d80 0x7fea4c0d42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fea34067b70 0x7fea3406a030 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea4c0df1a0 0x7fea4c0d3d70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 --2- 172.21.15.28:0/2779795914 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea4c050270 0x7fea4c04cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 >> 172.21.15.28:0/2779795914 conn(0x7fea4c054b00 msgr2=0x7fea4c007180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 shutdown_connections 2024-03-21T08:41:29.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.705+0000 7fea3f400640 1 -- 172.21.15.28:0/2779795914 wait complete. 2024-03-21T08:41:29.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.717+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f27680d2830 con 0x7f276815bd20 2024-03-21T08:41:29.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.717+0000 7f275e000640 1 -- 172.21.15.28:0/4094616351 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27600165e0 con 0x7f276815bd20 2024-03-21T08:41:29.724 INFO:teuthology.orchestra.run.smithi028.stdout:60129542152 2024-03-21T08:41:29.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.6 2024-03-21T08:41:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 msgr2=0x7f2744069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 0x7f2744069fe0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2758009eb0 tx=0x7f2758009190 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 msgr2=0x7f27680c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f276001ade0 tx=0x7f27600194d0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 shutdown_connections 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f276815bd20 0x7f27680c5e70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2744067b20 0x7f2744069fe0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27680cea00 0x7f27680cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 --2- 172.21.15.28:0/4094616351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27680b0e80 0x7f2768154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 >> 172.21.15.28:0/4094616351 conn(0x7f27680d1ab0 msgr2=0x7f2768073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 shutdown_connections 2024-03-21T08:41:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.725+0000 7f276cc00640 1 -- 172.21.15.28:0/4094616351 wait complete. 2024-03-21T08:41:29.736 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.5 2024-03-21T08:41:29.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.2 2024-03-21T08:41:29.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.0 2024-03-21T08:41:29.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.4 2024-03-21T08:41:29.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.757+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffa2c0d2830 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.757+0000 7ffa22a00640 1 -- 172.21.15.28:0/2124986392 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa1805a3d0 con 0x7ffa2c0c9bd0 2024-03-21T08:41:29.763 INFO:teuthology.orchestra.run.smithi028.stdout:60129542151 2024-03-21T08:41:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 msgr2=0x7ffa0c06a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 0x7ffa0c06a0b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0011a0 tx=0x7ffa1c004a50 comp rx=0 tx=0).stop 2024-03-21T08:41:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 msgr2=0x7ffa2c15a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 0x7ffa2c15a6f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e750 tx=0x7ffa1800ec10 comp rx=0 tx=0).stop 2024-03-21T08:41:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 shutdown_connections 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c0cb1e0 0x7ffa2c09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffa2c0c9bd0 0x7ffa2c15a6f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffa0c067bf0 0x7ffa0c06a0b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 --2- 172.21.15.28:0/2124986392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffa100039c0 0x7ffa2c0b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 >> 172.21.15.28:0/2124986392 conn(0x7ffa2c0d1ab0 msgr2=0x7ffa2c06fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 shutdown_connections 2024-03-21T08:41:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.761+0000 7ffa31c00640 1 -- 172.21.15.28:0/2124986392 wait complete. 2024-03-21T08:41:29.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.3 2024-03-21T08:41:29.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd48c0b4c80 con 0x7fd48c08a570 2024-03-21T08:41:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd482000640 1 -- 172.21.15.28:0/3926303028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd47c083e20 con 0x7fd48c08a570 2024-03-21T08:41:29.797 INFO:teuthology.orchestra.run.smithi028.stdout:64424509449 2024-03-21T08:41:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 msgr2=0x7fd46c06a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 0x7fd46c06a1e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4780ad000 tx=0x7fd4780a6000 comp rx=0 tx=0).stop 2024-03-21T08:41:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 msgr2=0x7fd48c1644a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00c530 tx=0x7fd47c00c9f0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 shutdown_connections 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd48c172960 0x7fd48c15ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd46c067d20 0x7fd46c06a1e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd48c172250 0x7fd48c16a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 --2- 172.21.15.28:0/3926303028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd48c08a570 0x7fd48c1644a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 >> 172.21.15.28:0/3926303028 conn(0x7fd48c06b770 msgr2=0x7fd48c0adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 shutdown_connections 2024-03-21T08:41:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.793+0000 7fd490a00640 1 -- 172.21.15.28:0/3926303028 wait complete. 2024-03-21T08:41:29.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.1 2024-03-21T08:41:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.813+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f86f0005860 con 0x7f871815e3a0 2024-03-21T08:41:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.813+0000 7f870f400640 1 -- 172.21.15.28:0/1831240511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8710056520 con 0x7f871815e3a0 2024-03-21T08:41:29.819 INFO:teuthology.orchestra.run.smithi028.stdout:64424509449 2024-03-21T08:41:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 msgr2=0x7f86f806a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 0x7f86f806a410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8704007fd0 tx=0x7f8704009040 comp rx=0 tx=0).stop 2024-03-21T08:41:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 msgr2=0x7f87180b5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 0x7f87180b5ad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f871000efc0 tx=0x7f871000c490 comp rx=0 tx=0).stop 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 shutdown_connections 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871815e3a0 0x7f87180b5ad0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f86f8067f50 0x7f86f806a410 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f87180d0c50 0x7f87180b8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 --2- 172.21.15.28:0/1831240511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f87180ced70 0x7f87181570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 >> 172.21.15.28:0/1831240511 conn(0x7f8718072aa0 msgr2=0x7f871815fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 shutdown_connections 2024-03-21T08:41:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:29.817+0000 7f871e200640 1 -- 172.21.15.28:0/1831240511 wait complete. 2024-03-21T08:41:29.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.7 2024-03-21T08:41:30.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-21T08:41:30.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-21T08:41:30.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-21T08:41:30.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-21T08:41:30.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-21T08:41:30.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-21T08:41:30.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-21T08:41:30.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-21T08:41:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.921+0000 7f5d13800640 1 Processor -- start 2024-03-21T08:41:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.921+0000 7f5d13800640 1 -- start start 2024-03-21T08:41:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d13800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d13800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d0c0cebd0 0x7f5d0c15deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d13800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0cf140 con 0x7f5d0c154350 2024-03-21T08:41:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d13800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c0cf420 con 0x7f5d0c0cebd0 2024-03-21T08:41:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d13800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c06df00 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48348/0 (socket says 172.21.15.28:48348) 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 -- 172.21.15.28:0/1935752783 learned_addr learned my addr 172.21.15.28:0/1935752783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 -- 172.21.15.28:0/1935752783 >> v2:172.21.15.28:3300/0 conn(0x7f5d0c0cebd0 msgr2=0x7f5d0c15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 --2- 172.21.15.28:0/1935752783 >> v2:172.21.15.28:3300/0 conn(0x7f5d0c0cebd0 0x7f5d0c15deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 -- 172.21.15.28:0/1935752783 >> v1:172.21.15.151:6789/0 conn(0x7f5d0c154350 legacy=0x7f5d0c0cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0be00640 1 -- 172.21.15.28:0/1935752783 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c06e1e0 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.925+0000 7f5d0b400640 1 -- 172.21.15.28:0/1935752783 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353403121 0 0) 0x7f5d0c0cf140 con 0x7f5d0c154350 2024-03-21T08:41:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d0be00640 1 --2- 172.21.15.28:0/1935752783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80091e0 tx=0x7f5cf801e1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f76b64ad799749b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d0b400640 1 -- 172.21.15.28:0/1935752783 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf80b3070 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d0b400640 1 -- 172.21.15.28:0/1935752783 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf80a6770 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d0b400640 1 -- 172.21.15.28:0/1935752783 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf80a6b40 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d13800640 1 -- 172.21.15.28:0/1935752783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 msgr2=0x7f5d0c15d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.933+0000 7f5d13800640 1 --2- 172.21.15.28:0/1935752783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80091e0 tx=0x7f5cf801e1f0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 -- 172.21.15.28:0/1935752783 shutdown_connections 2024-03-21T08:41:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 --2- 172.21.15.28:0/1935752783 >> v2:172.21.15.28:3300/0 conn(0x7f5d0c0cebd0 0x7f5d0c15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 --2- 172.21.15.28:0/1935752783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c15d970 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 -- 172.21.15.28:0/1935752783 >> 172.21.15.28:0/1935752783 conn(0x7f5d0c06b770 msgr2=0x7f5d0c0b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 -- 172.21.15.28:0/1935752783 shutdown_connections 2024-03-21T08:41:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.937+0000 7f5d13800640 1 -- 172.21.15.28:0/1935752783 wait complete. 2024-03-21T08:41:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 Processor -- start 2024-03-21T08:41:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 -- start start 2024-03-21T08:41:30.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c1f9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d0c0cebd0 0x7f5d0c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c06f580 con 0x7f5d0c154350 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c06f400 con 0x7f5d0c0cebd0 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.941+0000 7f5d13800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c06f700 con 0x7f5d0c0b2ad0 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55548/0 (socket says 172.21.15.28:55548) 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 -- 172.21.15.28:0/4245550903 learned_addr learned my addr 172.21.15.28:0/4245550903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d0be00640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d0c0cebd0 0x7f5d0c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d10e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c1f9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 -- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 msgr2=0x7f5d0c1f9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c1f9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 -- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d0c0cebd0 msgr2=0x7f5d0c152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d0c0cebd0 0x7f5d0c152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 -- 172.21.15.28:0/4245550903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c153000 con 0x7f5d0c154350 2024-03-21T08:41:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d11800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00b430 tx=0x7f5cfc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc015020 con 0x7f5d0c154350 2024-03-21T08:41:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc011830 con 0x7f5d0c154350 2024-03-21T08:41:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc011c20 con 0x7f5d0c154350 2024-03-21T08:41:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c155730 con 0x7f5d0c154350 2024-03-21T08:41:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.945+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c155cf0 con 0x7f5d0c154350 2024-03-21T08:41:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.953+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce40052a0 con 0x7f5d0c154350 2024-03-21T08:41:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.957+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5cfc018020 con 0x7f5d0c154350 2024-03-21T08:41:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.957+0000 7f5d08c00640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 0x7f5cec06a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.957+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5cfc0866d0 con 0x7f5d0c154350 2024-03-21T08:41:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.961+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5cfc01f020 con 0x7f5d0c154350 2024-03-21T08:41:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.961+0000 7f5d10e00640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 0x7f5cec06a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.969+0000 7f5d10e00640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 0x7f5cec06a0b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c2071d0 tx=0x7f5d04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.985+0000 7fef3aa00640 1 Processor -- start 2024-03-21T08:41:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.985+0000 7fef3aa00640 1 -- start start 2024-03-21T08:41:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef3aa00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 0x7fef3415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef3aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef3aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef34161a20 con 0x7fef340b31f0 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef3aa00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef340d1600 con 0x7fef340ced70 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef3aa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef340d2170 con 0x7fef340d0c50 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48366/0 (socket says 172.21.15.28:48366) 2024-03-21T08:41:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 -- 172.21.15.28:0/2730018834 learned_addr learned my addr 172.21.15.28:0/2730018834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef33e00640 1 --2- 172.21.15.28:0/2730018834 >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 0x7fef3415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 -- 172.21.15.28:0/2730018834 >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 msgr2=0x7fef3415e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 --2- 172.21.15.28:0/2730018834 >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 0x7fef3415e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 -- 172.21.15.28:0/2730018834 >> v1:172.21.15.28:6789/0 conn(0x7fef340b31f0 legacy=0x7fef340d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.989+0000 7fef38a00640 1 -- 172.21.15.28:0/2730018834 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3415e9e0 con 0x7fef340d0c50 2024-03-21T08:41:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.993+0000 7fef33e00640 1 --2- 172.21.15.28:0/2730018834 >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 0x7fef3415e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef38a00640 1 --2- 172.21.15.28:0/2730018834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009e50 tx=0x7fef2c01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aa55db0d2bf0526 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef32a00640 1 -- 172.21.15.28:0/2730018834 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0a5860 con 0x7fef340d0c50 2024-03-21T08:41:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef32a00640 1 -- 172.21.15.28:0/2730018834 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c0a5a00 con 0x7fef340d0c50 2024-03-21T08:41:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef32a00640 1 -- 172.21.15.28:0/2730018834 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0a5e10 con 0x7fef340d0c50 2024-03-21T08:41:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 -- 172.21.15.28:0/2730018834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 msgr2=0x7fef340d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 --2- 172.21.15.28:0/2730018834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009e50 tx=0x7fef2c01c9e0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 -- 172.21.15.28:0/2730018834 shutdown_connections 2024-03-21T08:41:31.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 --2- 172.21.15.28:0/2730018834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef340d10c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 --2- 172.21.15.28:0/2730018834 >> v2:172.21.15.151:3300/0 conn(0x7fef340ced70 0x7fef3415e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 -- 172.21.15.28:0/2730018834 >> 172.21.15.28:0/2730018834 conn(0x7fef34072aa0 msgr2=0x7fef34072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 -- 172.21.15.28:0/2730018834 shutdown_connections 2024-03-21T08:41:31.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:30.997+0000 7fef3aa00640 1 -- 172.21.15.28:0/2730018834 wait complete. 2024-03-21T08:41:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.001+0000 7fef3aa00640 1 Processor -- start 2024-03-21T08:41:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.001+0000 7fef3aa00640 1 -- start start 2024-03-21T08:41:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef340ced70 0x7fef3415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef33e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef33e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50854/0 (socket says 172.21.15.28:50854) 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef33e00640 1 -- 172.21.15.28:0/181686611 learned_addr learned my addr 172.21.15.28:0/181686611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef33400640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef340ced70 0x7fef3415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef3415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef340cf150 con 0x7fef340ced70 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef38a00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef3415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3415f280 con 0x7fef340b31f0 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.005+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef340cf2d0 con 0x7fef340d0c50 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 -- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 msgr2=0x7fef3415c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef3415c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 -- 172.21.15.28:0/181686611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef340ced70 msgr2=0x7fef3415bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef340ced70 0x7fef3415bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef340b7ec0 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef33e00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef2400d800 tx=0x7fef2400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24018020 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef240096c0 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef340b8160 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef340b8440 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24009ab0 con 0x7fef340b31f0 2024-03-21T08:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7f38e8600640 1 Processor -- start 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7f63a7200640 1 Processor -- start 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7f38e8600640 1 -- start start 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 0x7f38e015c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e8600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38e00d0990 0x7f38e00d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e8600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e015b7f0 con 0x7f38e015aae0 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e8600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e015b5d0 con 0x7f38e00d0990 2024-03-21T08:41:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00d1340 con 0x7f38e00cea70 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fef2401b020 con 0x7fef340b31f0 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.009+0000 7f63a7200640 1 -- start start 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a7200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a00eba20 0x7f63a00068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a00e7c00 con 0x7f63a00e7820 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a7200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a0057f30 con 0x7f63a0038290 2024-03-21T08:41:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a7200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0058210 con 0x7f63a00eba20 2024-03-21T08:41:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38e00d0990 0x7f38e00d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f38e00d0990 0x7f38e00d0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50868/0 (socket says 172.21.15.28:50868) 2024-03-21T08:41:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 -- 172.21.15.28:0/232624358 learned_addr learned my addr 172.21.15.28:0/232624358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3408afc0 con 0x7fef340b31f0 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e5200640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 0x7f38e015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 -- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 msgr2=0x7f38e015c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 0x7f38e015c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 -- 172.21.15.28:0/232624358 >> v1:172.21.15.151:6789/0 conn(0x7f38e015aae0 legacy=0x7f38e015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63a00e7820 0x7f63a00eb310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38560/0 (socket says 172.21.15.28:38560) 2024-03-21T08:41:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f63a5e00640 1 -- 172.21.15.28:0/3436422408 learned_addr learned my addr 172.21.15.28:0/3436422408 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e6600640 1 -- 172.21.15.28:0/232624358 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e015bad0 con 0x7f38e00d0990 2024-03-21T08:41:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7fef2be00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 0x7fef1406a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fef24086650 con 0x7fef340b31f0 2024-03-21T08:41:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7fef33400640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 0x7fef1406a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.013+0000 7f38e5200640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 0x7f38e015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:31.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7fef33400640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 0x7fef1406a0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef3415bb70 tx=0x7fef20009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598780897 0 0) 0x7f63a00e7c00 con 0x7f63a00e7820 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c003530 con 0x7f63a00e7820 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676929285 0 0) 0x7f638c003530 con 0x7f63a00e7820 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a00eba20 msgr2=0x7f63a00068c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a00eba20 0x7f63a00068c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 >> v1:172.21.15.151:6789/0 conn(0x7f63a0038290 legacy=0x7f63a00061b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00584f0 con 0x7f63a00e7820 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f6398001660 con 0x7f63a00e7820 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 >> v1:172.21.15.28:6789/0 conn(0x7f63a00e7820 legacy=0x7f63a00eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c003a70 0x7f638c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f638c004810 0x7f638c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f638c005320 0x7f638c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0058210 con 0x7f638c003a70 2024-03-21T08:41:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c00c280 con 0x7f638c004810 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c00c480 con 0x7f638c005320 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.017+0000 7f63a5e00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c003a70 0x7f638c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5400640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f638c004810 0x7f638c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a6800640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f638c005320 0x7f638c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38e6600640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e00d0990 0x7f38e00d0e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009870 tx=0x7f38d801c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b1fffb3371b2f0d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5e00640 1 -- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f638c005320 msgr2=0x7f638c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5e00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f638c005320 0x7f638c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5e00640 1 -- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f638c004810 msgr2=0x7f638c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5e00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f638c004810 0x7f638c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f63a5e00640 1 -- 172.21.15.28:0/3436422408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c003530 con 0x7f638c003a70 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38dfe00640 1 -- 172.21.15.28:0/232624358 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80a7070 con 0x7f38e00d0990 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38dfe00640 1 -- 172.21.15.28:0/232624358 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d80b0070 con 0x7f38e00d0990 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38dfe00640 1 -- 172.21.15.28:0/232624358 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80ab890 con 0x7f38e00d0990 2024-03-21T08:41:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fef24022020 con 0x7fef340b31f0 2024-03-21T08:41:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38e8600640 1 -- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e00d0990 msgr2=0x7f38e00d0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38e8600640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e00d0990 0x7f38e00d0e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009870 tx=0x7f38d801c490 comp rx=0 tx=0).stop 2024-03-21T08:41:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.021+0000 7f38e8600640 1 -- 172.21.15.28:0/232624358 shutdown_connections 2024-03-21T08:41:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f38e8600640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e00d0990 0x7f38e00d0e00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f38e8600640 1 --2- 172.21.15.28:0/232624358 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00cea70 0x7f38e015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f38e8600640 1 -- 172.21.15.28:0/232624358 >> 172.21.15.28:0/232624358 conn(0x7f38e0099a30 msgr2=0x7f38e0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 Processor -- start 2024-03-21T08:41:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 -- start start 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a5e00640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c003a70 0x7f638c003e50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63980050e0 tx=0x7f639801d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29516f371d73f894 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398004ad0 con 0x7f638c003a70 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398005b00 con 0x7f638c003a70 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a4a00640 1 -- 172.21.15.28:0/3436422408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398008110 con 0x7f638c003a70 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f47f00c3920 0x7f47f006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a7200640 1 -- 172.21.15.28:0/3436422408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c003a70 msgr2=0x7f638c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f63a7200640 1 --2- 172.21.15.28:0/3436422408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c003a70 0x7f638c003e50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63980050e0 tx=0x7f639801d350 comp rx=0 tx=0).stop 2024-03-21T08:41:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f38e8600640 1 -- 172.21.15.28:0/232624358 shutdown_connections 2024-03-21T08:41:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f38e8600640 1 -- 172.21.15.28:0/232624358 wait complete. 2024-03-21T08:41:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0166d00 con 0x7f47f006b170 2024-03-21T08:41:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f0166870 con 0x7f47f00c3920 2024-03-21T08:41:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.025+0000 7f47f6200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00c4240 con 0x7f47f00afcf0 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47f006b170 0x7f47f006b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38572/0 (socket says 172.21.15.28:38572) 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47efe00640 1 -- 172.21.15.28:0/1898269198 learned_addr learned my addr 172.21.15.28:0/1898269198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ef400640 1 --2- 172.21.15.28:0/1898269198 >> v2:172.21.15.151:3300/0 conn(0x7f47f00c3920 0x7f47f006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47eea00640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ef400640 1 -- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 msgr2=0x7f47f00c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ee000640 1 -- 172.21.15.28:0/1898269198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153477586 0 0) 0x7f47f0166d00 con 0x7f47f006b170 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ef400640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ef400640 1 -- 172.21.15.28:0/1898269198 >> v1:172.21.15.28:6789/0 conn(0x7f47f006b170 legacy=0x7f47f006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47ef400640 1 -- 172.21.15.28:0/1898269198 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0165f40 con 0x7f47f00c3920 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f47eea00640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f63a6800640 1 -- 172.21.15.28:0/3436422408 reap_dead start 2024-03-21T08:41:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f38e8600640 1 Processor -- start 2024-03-21T08:41:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f63a7200640 1 -- 172.21.15.28:0/3436422408 shutdown_connections 2024-03-21T08:41:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f63a7200640 1 -- 172.21.15.28:0/3436422408 >> 172.21.15.28:0/3436422408 conn(0x7f63a00559f0 msgr2=0x7f63a0055e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f63a7200640 1 -- 172.21.15.28:0/3436422408 shutdown_connections 2024-03-21T08:41:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.029+0000 7f63a7200640 1 -- 172.21.15.28:0/3436422408 wait complete. 2024-03-21T08:41:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f63a7200640 1 Processor -- start 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f63a7200640 1 -- start start 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 -- start start 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 0x7f38e006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 0x7f38e0159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 0x7f38e0153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 0x7f38e006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 0x7f38e006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55570/0 (socket says 172.21.15.28:55570) 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e5200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 0x7f38e0159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e5200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 0x7f38e0159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48428/0 (socket says 172.21.15.28:48428) 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f47ef400640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f00c3920 0x7f47f006eb40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f47e00097c0 tx=0x7f47e001c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4964d6c5510e35d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f47ee000640 1 -- 172.21.15.28:0/1898269198 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00b0070 con 0x7f47f00c3920 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f47ee000640 1 -- 172.21.15.28:0/1898269198 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e00ab600 con 0x7f47f00c3920 2024-03-21T08:41:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f47ee000640 1 -- 172.21.15.28:0/1898269198 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00b0070 con 0x7f47f00c3920 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00b1d70 con 0x7f38e00cea70 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00b1bf0 con 0x7f38e015aae0 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00cd500 con 0x7f38e00d0990 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e5c00640 1 -- 172.21.15.28:0/3051465901 learned_addr learned my addr 172.21.15.28:0/3051465901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.033+0000 7f38e6600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 0x7f38e0153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63a004ed60 0x7f63a0039920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a000d560 0x7f63a000bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00ebe90 con 0x7f63a0009ca0 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0054700 con 0x7f63a004ed60 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a7200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a003ed50 con 0x7f63a000d560 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 -- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f00c3920 msgr2=0x7f47f006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f00c3920 0x7f47f006eb40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f47e00097c0 tx=0x7f47e001c380 comp rx=0 tx=0).stop 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 -- 172.21.15.28:0/1898269198 shutdown_connections 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 --2- 172.21.15.28:0/1898269198 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f00c3920 0x7f47f006eb40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 -- 172.21.15.28:0/1898269198 >> 172.21.15.28:0/1898269198 conn(0x7f47f0099a30 msgr2=0x7f47f0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 -- 172.21.15.28:0/1898269198 shutdown_connections 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55572/0 (socket says 172.21.15.28:55572) 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 -- 172.21.15.28:0/1086271699 learned_addr learned my addr 172.21.15.28:0/1086271699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a5e00640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63a004ed60 0x7f63a0039920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f47f6200640 1 -- 172.21.15.28:0/1898269198 wait complete. 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 Processor -- start 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 -- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a000d560 msgr2=0x7f63a000bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a000d560 0x7f63a000bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 -- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63a004ed60 msgr2=0x7f63a0039920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63a004ed60 0x7f63a0039920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 -- 172.21.15.28:0/1086271699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a000c4c0 con 0x7f63a0009ca0 2024-03-21T08:41:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f63a6800640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f63a802d610 tx=0x7f63a80684e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 -- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 msgr2=0x7f38e0159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 0x7f38e0159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 -- 172.21.15.28:0/3051465901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 msgr2=0x7f38e006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 0x7f38e006ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 -- 172.21.15.28:0/3051465901 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00c7f60 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e6600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 0x7f38e0153b30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38d80ac000 tx=0x7f38d801acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80be5f0 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e00c8240 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d80be790 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80be5f0 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.037+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e00c8800 con 0x7f38e015aae0 2024-03-21T08:41:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e00d1120 con 0x7f38e015aae0 2024-03-21T08:41:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 -- start start 2024-03-21T08:41:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8073f40 con 0x7f63a0009ca0 2024-03-21T08:41:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a80740e0 con 0x7f63a0009ca0 2024-03-21T08:41:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8073080 con 0x7f63a0009ca0 2024-03-21T08:41:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 0x7f47f00b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f38d80b43f0 con 0x7f38e015aae0 2024-03-21T08:41:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f38dd600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 0x7f38c006a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 0x7f47f0164310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47eea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f38e5c00640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 0x7f38c006a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47eea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48450/0 (socket says 172.21.15.28:48450) 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f016faf0 con 0x7f47f00c3920 2024-03-21T08:41:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f016f970 con 0x7f47f006b170 2024-03-21T08:41:31.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47f6200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f016fc70 con 0x7f47f00afcf0 2024-03-21T08:41:31.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f38d811d890 con 0x7f38e015aae0 2024-03-21T08:41:31.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47eea00640 1 -- 172.21.15.28:0/3508166851 learned_addr learned my addr 172.21.15.28:0/3508166851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f63a7200640 1 -- 172.21.15.28:0/1086271699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a000c7a0 con 0x7f63a0009ca0 2024-03-21T08:41:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f47efe00640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 0x7f47f0164310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 0x7f47f00b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.041+0000 7f63a7200640 1 -- 172.21.15.28:0/1086271699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a000cd60 con 0x7f63a0009ca0 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f38e5c00640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 0x7f38c006a1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009f50 tx=0x7f38d4009590 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 -- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 msgr2=0x7f47f00cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 -- 172.21.15.28:0/3508166851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 msgr2=0x7f47f0164310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 0x7f47f0164310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 -- 172.21.15.28:0/3508166851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00b04d0 con 0x7f47f006b170 2024-03-21T08:41:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47efe00640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 0x7f47f0164310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c0052a0 con 0x7f63a0009ca0 2024-03-21T08:41:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47ef400640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 0x7f47f00b37a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b3540 tx=0x7f47e0019120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f63a8073220 con 0x7f63a0009ca0 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.045+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00b9020 con 0x7f47f006b170 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e00abbc0 con 0x7f47f006b170 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00b9020 con 0x7f47f006b170 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f00b3ea0 con 0x7f47f006b170 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f00b4460 con 0x7f47f006b170 2024-03-21T08:41:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f38d8016110 con 0x7f38e015aae0 2024-03-21T08:41:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f639e000640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 0x7f638806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f63a81006c0 con 0x7f63a0009ca0 2024-03-21T08:41:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f009ea60 con 0x7f47f006b170 2024-03-21T08:41:31.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.049+0000 7f63a5e00640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 0x7f638806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.053+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f47e00bc020 con 0x7f47f006b170 2024-03-21T08:41:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.053+0000 7f47e7400640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 0x7f47d006a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.053+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f47e011ff10 con 0x7f47f006b170 2024-03-21T08:41:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.053+0000 7f47eea00640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 0x7f47d006a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.053+0000 7f63a5e00640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 0x7f638806a010 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f63980ab000 tx=0x7f63980b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.057+0000 7f47eea00640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 0x7f47d006a1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009580 tx=0x7f47dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.057+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f47e00c3030 con 0x7f47f006b170 2024-03-21T08:41:31.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.057+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f63a8099020 con 0x7f63a0009ca0 2024-03-21T08:41:31.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 Processor -- start 2024-03-21T08:41:31.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 -- start start 2024-03-21T08:41:31.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a7415dc40 0x7f5a7406d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7406dd50 con 0x7f5a74071e70 2024-03-21T08:41:31.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a740cc690 con 0x7f5a740b0b70 2024-03-21T08:41:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.093+0000 7f5a73e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a740ce4d0 con 0x7f5a7415dc40 2024-03-21T08:41:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a7415dc40 0x7f5a7406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a7415dc40 0x7f5a7406d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50932/0 (socket says 172.21.15.28:50932) 2024-03-21T08:41:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 -- 172.21.15.28:0/1101378152 learned_addr learned my addr 172.21.15.28:0/1101378152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 -- 172.21.15.28:0/1101378152 >> v1:172.21.15.151:6789/0 conn(0x7f5a740b0b70 legacy=0x7f5a74071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a71600640 1 -- 172.21.15.28:0/1101378152 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273550055 0 0) 0x7f5a740cc690 con 0x7f5a740b0b70 2024-03-21T08:41:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 -- 172.21.15.28:0/1101378152 >> v1:172.21.15.28:6789/0 conn(0x7f5a74071e70 legacy=0x7f5a74167800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.097+0000 7f5a72a00640 1 -- 172.21.15.28:0/1101378152 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7406dfd0 con 0x7f5a7415dc40 2024-03-21T08:41:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.101+0000 7f5a72a00640 1 --2- 172.21.15.28:0/1101378152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a7415dc40 0x7f5a7406d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a6401c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e41cb7e4a27e4f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.101+0000 7f361c800640 1 Processor -- start 2024-03-21T08:41:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 -- start start 2024-03-21T08:41:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3614163440 0x7f3614071330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3614162710 0x7f36140b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614071aa0 con 0x7f36140b5070 2024-03-21T08:41:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3614071870 con 0x7f3614162710 2024-03-21T08:41:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361c800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3614071d80 con 0x7f3614163440 2024-03-21T08:41:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f5a71600640 1 -- 172.21.15.28:0/1101378152 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640a95f0 con 0x7f5a7415dc40 2024-03-21T08:41:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f5a71600640 1 -- 172.21.15.28:0/1101378152 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a640a9790 con 0x7f5a7415dc40 2024-03-21T08:41:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f5a73e00640 1 -- 172.21.15.28:0/1101378152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a7415dc40 msgr2=0x7f5a7406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f3619e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3614163440 0x7f3614071330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f36140b5070 0x7f3614165c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38600/0 (socket says 172.21.15.28:38600) 2024-03-21T08:41:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f361a800640 1 -- 172.21.15.28:0/2999759013 learned_addr learned my addr 172.21.15.28:0/2999759013 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f5a71600640 1 -- 172.21.15.28:0/1101378152 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a640a95f0 con 0x7f5a7415dc40 2024-03-21T08:41:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f5a73e00640 1 --2- 172.21.15.28:0/1101378152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a7415dc40 0x7f5a7406d810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a6401c530 comp rx=0 tx=0).stop 2024-03-21T08:41:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.105+0000 7f3619400640 1 --2- 172.21.15.28:0/2999759013 >> v2:172.21.15.28:3300/0 conn(0x7f3614162710 0x7f36140b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f5a73e00640 1 -- 172.21.15.28:0/1101378152 shutdown_connections 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f5a73e00640 1 --2- 172.21.15.28:0/1101378152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a7415dc40 0x7f5a7406d810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f5a73e00640 1 -- 172.21.15.28:0/1101378152 >> 172.21.15.28:0/1101378152 conn(0x7f5a740b14a0 msgr2=0x7f5a740b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f3619e00640 1 -- 172.21.15.28:0/2999759013 >> v2:172.21.15.28:3300/0 conn(0x7f3614162710 msgr2=0x7f36140b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f3619e00640 1 --2- 172.21.15.28:0/2999759013 >> v2:172.21.15.28:3300/0 conn(0x7f3614162710 0x7f36140b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f3619e00640 1 -- 172.21.15.28:0/2999759013 >> v1:172.21.15.28:6789/0 conn(0x7f36140b5070 legacy=0x7f3614165c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f3619e00640 1 -- 172.21.15.28:0/2999759013 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614072060 con 0x7f3614163440 2024-03-21T08:41:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.109+0000 7f5a73e00640 1 -- 172.21.15.28:0/1101378152 shutdown_connections 2024-03-21T08:41:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f5a73e00640 1 -- 172.21.15.28:0/1101378152 wait complete. 2024-03-21T08:41:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f5a73e00640 1 Processor -- start 2024-03-21T08:41:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f3619e00640 1 --2- 172.21.15.28:0/2999759013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3614163440 0x7f3614071330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36140504d0 tx=0x7f360801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b883036860a20bc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f3618a00640 1 -- 172.21.15.28:0/2999759013 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080b0070 con 0x7f3614163440 2024-03-21T08:41:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f5a73e00640 1 -- start start 2024-03-21T08:41:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74071e70 0x7f5a7415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f3618a00640 1 -- 172.21.15.28:0/2999759013 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080ab600 con 0x7f3614163440 2024-03-21T08:41:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.113+0000 7f3618a00640 1 -- 172.21.15.28:0/2999759013 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080ab9d0 con 0x7f3614163440 2024-03-21T08:41:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a740b0b70 0x7f5a740b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7415e020 con 0x7f5a74071e70 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406eb50 con 0x7f5a740b0b70 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406f070 con 0x7f5a740b4c00 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48464/0 (socket says 172.21.15.28:48464) 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 -- 172.21.15.28:0/2290184904 learned_addr learned my addr 172.21.15.28:0/2290184904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a72a00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74071e70 0x7f5a7415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a72000640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a740b0b70 0x7f5a740b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 -- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a740b0b70 msgr2=0x7f5a740b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a740b0b70 0x7f5a740b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 -- 172.21.15.28:0/2290184904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74071e70 msgr2=0x7f5a7415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74071e70 0x7f5a7415c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 -- 172.21.15.28:0/2290184904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a740b5140 con 0x7f5a740b4c00 2024-03-21T08:41:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73400640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c730 tx=0x7f5a6c00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c016020 con 0x7f5a740b4c00 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c006d80 con 0x7f5a740b4c00 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c003bc0 con 0x7f5a740b4c00 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 -- 172.21.15.28:0/2290184904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7414bf20 con 0x7f5a740b4c00 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f5a73e00640 1 -- 172.21.15.28:0/2290184904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7414c4e0 con 0x7f5a740b4c00 2024-03-21T08:41:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7404f410 con 0x7f5a740b4c00 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f361c800640 1 -- 172.21.15.28:0/2999759013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3614163440 msgr2=0x7f3614071330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.117+0000 7f361c800640 1 --2- 172.21.15.28:0/2999759013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3614163440 0x7f3614071330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36140504d0 tx=0x7f360801c480 comp rx=0 tx=0).stop 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 -- 172.21.15.28:0/2999759013 shutdown_connections 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 --2- 172.21.15.28:0/2999759013 >> v2:172.21.15.28:3300/0 conn(0x7f3614162710 0x7f36140b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 --2- 172.21.15.28:0/2999759013 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3614163440 0x7f3614071330 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 -- 172.21.15.28:0/2999759013 >> 172.21.15.28:0/2999759013 conn(0x7f36140b3520 msgr2=0x7f36140b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 -- 172.21.15.28:0/2999759013 shutdown_connections 2024-03-21T08:41:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f361c800640 1 -- 172.21.15.28:0/2999759013 wait complete. 2024-03-21T08:41:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.121+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5a6c00a850 con 0x7f5a740b4c00 2024-03-21T08:41:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f5a6aa00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 0x7f5a5406a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5a6c085ed0 con 0x7f5a740b4c00 2024-03-21T08:41:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 Processor -- start 2024-03-21T08:41:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 -- start start 2024-03-21T08:41:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f5a72a00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 0x7f5a5406a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f5a72a00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 0x7f5a5406a0b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a640ae000 tx=0x7f5a64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5a6c020020 con 0x7f5a740b4c00 2024-03-21T08:41:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 0x7f36140ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f3619e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f3619e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50958/0 (socket says 172.21.15.28:50958) 2024-03-21T08:41:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 0x7f361414d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361416d3b0 con 0x7f3614163440 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361416d230 con 0x7f36140b5070 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f361c800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361416d530 con 0x7f3614162710 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f3619400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 0x7f36140ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f3619400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 0x7f36140ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48476/0 (socket says 172.21.15.28:48476) 2024-03-21T08:41:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.125+0000 7f3619400640 1 -- 172.21.15.28:0/1417261263 learned_addr learned my addr 172.21.15.28:0/1417261263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.129+0000 7f361a800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 0x7f361414d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.129+0000 7f3619e00640 1 -- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 msgr2=0x7f36140ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.129+0000 7f3619e00640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 0x7f36140ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.129+0000 7f3619e00640 1 -- 172.21.15.28:0/1417261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 msgr2=0x7f361414d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.133+0000 7f3619e00640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 0x7f361414d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.133+0000 7f3619e00640 1 -- 172.21.15.28:0/1417261263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36140cea60 con 0x7f36140b5070 2024-03-21T08:41:31.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.133+0000 7f361a800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 0x7f361414d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.133+0000 7f3619e00640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f36080ab000 tx=0x7f360801acc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080b0050 con 0x7f36140b5070 2024-03-21T08:41:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080bc070 con 0x7f36140b5070 2024-03-21T08:41:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080b5830 con 0x7f36140b5070 2024-03-21T08:41:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36140ced00 con 0x7f36140b5070 2024-03-21T08:41:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36140cf2c0 con 0x7f36140b5070 2024-03-21T08:41:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.137+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361404f410 con 0x7f36140b5070 2024-03-21T08:41:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.141+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f36080b59d0 con 0x7f36140b5070 2024-03-21T08:41:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.141+0000 7f360e000640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 0x7f35f806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.141+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3608120830 con 0x7f36140b5070 2024-03-21T08:41:31.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.145+0000 7f3619400640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 0x7f35f806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.145+0000 7f3619400640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 0x7f35f806a010 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36140b57e0 tx=0x7f3604009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.145+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3608016090 con 0x7f36140b5070 2024-03-21T08:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.149+0000 7f1158200640 1 Processor -- start 2024-03-21T08:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.149+0000 7f1158200640 1 -- start start 2024-03-21T08:41:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1158200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f115015daf0 0x7f11500cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1158200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11500ae5d0 0x7f11500aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1158200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115006e1b0 con 0x7f115015ded0 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1158200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11500aef50 con 0x7f115015daf0 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1158200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1150071780 con 0x7f11500ae5d0 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f115015daf0 0x7f11500cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f115015daf0 0x7f11500cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50972/0 (socket says 172.21.15.28:50972) 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 -- 172.21.15.28:0/3985922463 learned_addr learned my addr 172.21.15.28:0/3985922463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1154e00640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11500ae5d0 0x7f11500aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 -- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11500ae5d0 msgr2=0x7f11500aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11500ae5d0 0x7f11500aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 -- 172.21.15.28:0/3985922463 >> v1:172.21.15.151:6789/0 conn(0x7f115015ded0 legacy=0x7f115015e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.153+0000 7f1155800640 1 -- 172.21.15.28:0/3985922463 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150071a60 con 0x7f115015daf0 2024-03-21T08:41:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.157+0000 7f1155800640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f11500cfc00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1144009870 tx=0x7f114401c570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9dd7a90aff44a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.157+0000 7f114fe00640 1 -- 172.21.15.28:0/3985922463 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440a95f0 con 0x7f115015daf0 2024-03-21T08:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.157+0000 7f114fe00640 1 -- 172.21.15.28:0/3985922463 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11440a9790 con 0x7f115015daf0 2024-03-21T08:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.157+0000 7f114fe00640 1 -- 172.21.15.28:0/3985922463 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440a9b60 con 0x7f115015daf0 2024-03-21T08:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 -- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 msgr2=0x7f11500cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f11500cfc00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1144009870 tx=0x7f114401c570 comp rx=0 tx=0).stop 2024-03-21T08:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 -- 172.21.15.28:0/3985922463 shutdown_connections 2024-03-21T08:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11500ae5d0 0x7f11500aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 --2- 172.21.15.28:0/3985922463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f11500cfc00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 -- 172.21.15.28:0/3985922463 >> 172.21.15.28:0/3985922463 conn(0x7f11500674f0 msgr2=0x7f115006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 -- 172.21.15.28:0/3985922463 shutdown_connections 2024-03-21T08:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.161+0000 7f1158200640 1 -- 172.21.15.28:0/3985922463 wait complete. 2024-03-21T08:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 Processor -- start 2024-03-21T08:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 -- start start 2024-03-21T08:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11500ae5d0 0x7f1150204680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 0x7f11500aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1154e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1154e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:50986/0 (socket says 172.21.15.28:50986) 2024-03-21T08:41:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1154e00640 1 -- 172.21.15.28:0/3361790257 learned_addr learned my addr 172.21.15.28:0/3361790257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500d0140 con 0x7f11500ae5d0 2024-03-21T08:41:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1156200640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 0x7f11500aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1155800640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11500ae5d0 0x7f1150204680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500725e0 con 0x7f115015daf0 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.165+0000 7f1158200640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11500ae100 con 0x7f115015ded0 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 -- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 msgr2=0x7f11500aa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 0x7f11500aa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 -- 172.21.15.28:0/3361790257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11500ae5d0 msgr2=0x7f1150204680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11500ae5d0 0x7f1150204680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500aaed0 con 0x7f115015daf0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1154e00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f114800efc0 tx=0x7f114800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1156200640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 0x7f11500aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148018020 con 0x7f115015daf0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1148014480 con 0x7f115015daf0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148014830 con 0x7f115015daf0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1158200640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11500ab1b0 con 0x7f115015daf0 2024-03-21T08:41:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1158200640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150204bc0 con 0x7f115015daf0 2024-03-21T08:41:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.169+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115004f410 con 0x7f115015daf0 2024-03-21T08:41:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.173+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f114801b020 con 0x7f115015daf0 2024-03-21T08:41:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.173+0000 7f114d600640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 0x7f113006a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.173+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1148087780 con 0x7f115015daf0 2024-03-21T08:41:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.173+0000 7f1155800640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 0x7f113006a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.177+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1148022070 con 0x7f115015daf0 2024-03-21T08:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.181+0000 7f1155800640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 0x7f113006a030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f11440ae000 tx=0x7f11440b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.309+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5ce4005860 con 0x7f5d0c154350 2024-03-21T08:41:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.317+0000 7f5d08c00640 1 -- 172.21.15.28:0/4245550903 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5cfc056d80 con 0x7f5d0c154350 2024-03-21T08:41:31.320 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 msgr2=0x7f5cec06a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 0x7f5cec06a0b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c2071d0 tx=0x7f5d04009040 comp rx=0 tx=0).stop 2024-03-21T08:41:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 msgr2=0x7f5d0c1567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00b430 tx=0x7f5cfc00b8f0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 shutdown_connections 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c154350 0x7f5d0c1567d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5d0c0cebd0 0x7f5d0c152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cec067bf0 0x7f5cec06a0b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 --2- 172.21.15.28:0/4245550903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5d0c0b2ad0 0x7f5d0c1f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 >> 172.21.15.28:0/4245550903 conn(0x7f5d0c06b770 msgr2=0x7f5d0c15d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 shutdown_connections 2024-03-21T08:41:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.321+0000 7f5d13800640 1 -- 172.21.15.28:0/4245550903 wait complete. 2024-03-21T08:41:31.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.349+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fef3404f4f0 con 0x7fef340b31f0 2024-03-21T08:41:31.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.6 2024-03-21T08:41:31.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef2be00640 1 -- 172.21.15.28:0/181686611 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef24056d90 con 0x7fef340b31f0 2024-03-21T08:41:31.358 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:31.358 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 msgr2=0x7fef1406a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 0x7fef1406a0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef3415bb70 tx=0x7fef20009060 comp rx=0 tx=0).stop 2024-03-21T08:41:31.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 msgr2=0x7fef340ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef2400d800 tx=0x7fef2400dcc0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 shutdown_connections 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef14067bf0 0x7fef1406a0b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef340d0c50 0x7fef3415c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef340ced70 0x7fef3415bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 --2- 172.21.15.28:0/181686611 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef340b31f0 0x7fef340ca380 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 >> 172.21.15.28:0/181686611 conn(0x7fef34072aa0 msgr2=0x7fef3415b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 shutdown_connections 2024-03-21T08:41:31.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.353+0000 7fef3aa00640 1 -- 172.21.15.28:0/181686611 wait complete. 2024-03-21T08:41:31.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.5 2024-03-21T08:41:31.382 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.401+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f38e009eb30 con 0x7f38e015aae0 2024-03-21T08:41:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.401+0000 7f38dd600640 1 -- 172.21.15.28:0/3051465901 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38d8016110 con 0x7f38e015aae0 2024-03-21T08:41:31.406 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.401+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f637c005860 con 0x7f63a0009ca0 2024-03-21T08:41:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.401+0000 7f639e000640 1 -- 172.21.15.28:0/1086271699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f63a80d0e20 con 0x7f63a0009ca0 2024-03-21T08:41:31.407 INFO:teuthology.orchestra.run.smithi028.stdout:60129542154 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 msgr2=0x7f38c006a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 0x7f38c006a1e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38d4009f50 tx=0x7f38d4009590 comp rx=0 tx=0).stop 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 msgr2=0x7f38e0153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 0x7f38e0153b30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38d80ac000 tx=0x7f38d801acd0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 shutdown_connections 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38e015aae0 0x7f38e0153b30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38c0067d20 0x7f38c006a1e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f38e00d0990 0x7f38e0159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 --2- 172.21.15.28:0/3051465901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e00cea70 0x7f38e006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 >> 172.21.15.28:0/3051465901 conn(0x7f38e0099a30 msgr2=0x7f38e0099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 shutdown_connections 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f38e8600640 1 -- 172.21.15.28:0/3051465901 wait complete. 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 msgr2=0x7f638806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 0x7f638806a010 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f63980ab000 tx=0x7f63980b2040 comp rx=0 tx=0).stop 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 msgr2=0x7f63a000a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f63a802d610 tx=0x7f63a80684e0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 shutdown_connections 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0009ca0 0x7f63a000a120 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6388067b50 0x7f638806a010 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f63980ab000 tx=0x7f63980b2040 comp rx=0 tx=0).stop 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f63a000d560 0x7f63a000bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 --2- 172.21.15.28:0/1086271699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f63a004ed60 0x7f63a0039920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 >> 172.21.15.28:0/1086271699 conn(0x7f63a00559f0 msgr2=0x7f63a00e9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f47f004f4f0 con 0x7f47f006b170 2024-03-21T08:41:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 shutdown_connections 2024-03-21T08:41:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f6393400640 1 -- 172.21.15.28:0/1086271699 wait complete. 2024-03-21T08:41:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.405+0000 7f47e7400640 1 -- 172.21.15.28:0/3508166851 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47e00f6030 con 0x7f47f006b170 2024-03-21T08:41:31.412 INFO:teuthology.orchestra.run.smithi028.stdout:64424509449 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 msgr2=0x7f47d006a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 0x7f47d006a1e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009580 tx=0x7f47dc009190 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 msgr2=0x7f47f00b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 0x7f47f00b37a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b3540 tx=0x7f47e0019120 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 shutdown_connections 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47f00c3920 0x7f47f0164310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47d0067d20 0x7f47d006a1e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f47f00afcf0 0x7f47f00cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 --2- 172.21.15.28:0/3508166851 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f47f006b170 0x7f47f00b37a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 >> 172.21.15.28:0/3508166851 conn(0x7f47f0099a30 msgr2=0x7f47f00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 shutdown_connections 2024-03-21T08:41:31.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.409+0000 7f47f6200640 1 -- 172.21.15.28:0/3508166851 wait complete. 2024-03-21T08:41:31.437 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.2 2024-03-21T08:41:31.438 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.4 2024-03-21T08:41:31.445 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.0 2024-03-21T08:41:31.447 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.457+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5a740ca6d0 con 0x7f5a740b4c00 2024-03-21T08:41:31.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a6aa00640 1 -- 172.21.15.28:0/2290184904 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a7406dd50 con 0x7f5a740b4c00 2024-03-21T08:41:31.463 INFO:teuthology.orchestra.run.smithi028.stdout:60129542153 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 msgr2=0x7f5a5406a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 0x7f5a5406a0b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a640ae000 tx=0x7f5a64008040 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 msgr2=0x7f5a74156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c730 tx=0x7f5a6c00cbf0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 shutdown_connections 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a54067bf0 0x7f5a5406a0b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a740b4c00 0x7f5a74156760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a740b0b70 0x7f5a740b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 --2- 172.21.15.28:0/2290184904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a74071e70 0x7f5a7415c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 >> 172.21.15.28:0/2290184904 conn(0x7f5a740b14a0 msgr2=0x7f5a740b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 shutdown_connections 2024-03-21T08:41:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.461+0000 7f5a5fe00640 1 -- 172.21.15.28:0/2290184904 wait complete. 2024-03-21T08:41:31.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.465+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f36140b43c0 con 0x7f36140b5070 2024-03-21T08:41:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.465+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f115006c740 con 0x7f115015daf0 2024-03-21T08:41:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.465+0000 7f360e000640 1 -- 172.21.15.28:0/1417261263 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f36080f0ef0 con 0x7f36140b5070 2024-03-21T08:41:31.469 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.465+0000 7f114d600640 1 -- 172.21.15.28:0/3361790257 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1148010040 con 0x7f115015daf0 2024-03-21T08:41:31.470 INFO:teuthology.orchestra.run.smithi028.stdout:64424509450 2024-03-21T08:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 msgr2=0x7f35f806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 0x7f35f806a010 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f36140b57e0 tx=0x7f3604009190 comp rx=0 tx=0).stop 2024-03-21T08:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 msgr2=0x7f113006a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 0x7f113006a030 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f11440ae000 tx=0x7f11440b3040 comp rx=0 tx=0).stop 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 msgr2=0x7f1150153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f114800efc0 tx=0x7f114800c490 comp rx=0 tx=0).stop 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 msgr2=0x7f361414a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f36080ab000 tx=0x7f360801acc0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 shutdown_connections 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3614163440 0x7f361414d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f35f8067b50 0x7f35f806a010 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3614162710 0x7f36140ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 shutdown_connections 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1130067b70 0x7f113006a030 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f115015ded0 0x7f11500aa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f115015daf0 0x7f1150153070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 --2- 172.21.15.28:0/1417261263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36140b5070 0x7f361414a200 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 >> 172.21.15.28:0/1417261263 conn(0x7f36140b3520 msgr2=0x7f361414efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 --2- 172.21.15.28:0/3361790257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11500ae5d0 0x7f1150204680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 >> 172.21.15.28:0/3361790257 conn(0x7f11500674f0 msgr2=0x7f1150153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 shutdown_connections 2024-03-21T08:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f361c800640 1 -- 172.21.15.28:0/1417261263 wait complete. 2024-03-21T08:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 shutdown_connections 2024-03-21T08:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.469+0000 7f1142a00640 1 -- 172.21.15.28:0/3361790257 wait complete. 2024-03-21T08:41:31.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.3 2024-03-21T08:41:31.490 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.1 2024-03-21T08:41:31.495 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.7 2024-03-21T08:41:31.497 DEBUG:teuthology.parallel:result is None 2024-03-21T08:41:31.497 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-21T08:41:31.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:41:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 Processor -- start 2024-03-21T08:41:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 -- start start 2024-03-21T08:41:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f51580af060 0x7f51580cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b34f0 0x7f51580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f51580b5350 0x7f5158071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51580ce6e0 con 0x7f51580af060 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515815bf00 con 0x7f51580b5350 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515fa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51580cdad0 con 0x7f51580b34f0 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f51580af060 0x7f51580cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f51580af060 0x7f51580cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55624/0 (socket says 172.21.15.28:55624) 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.677+0000 7f515d000640 1 -- 172.21.15.28:0/3236895080 learned_addr learned my addr 172.21.15.28:0/3236895080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f514fe00640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b34f0 0x7f51580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515da00640 1 --2- 172.21.15.28:0/3236895080 >> v2:172.21.15.151:3300/0 conn(0x7f51580b5350 0x7f5158071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 -- 172.21.15.28:0/3236895080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b34f0 msgr2=0x7f51580b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b34f0 0x7f51580b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 -- 172.21.15.28:0/3236895080 >> v2:172.21.15.151:3300/0 conn(0x7f51580b5350 msgr2=0x7f5158071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 --2- 172.21.15.28:0/3236895080 >> v2:172.21.15.151:3300/0 conn(0x7f51580b5350 0x7f5158071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 -- 172.21.15.28:0/3236895080 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580cddb0 con 0x7f51580af060 2024-03-21T08:41:31.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f515d000640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580af060 0x7f51580cd380 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f51580cc140 tx=0x7f514801c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec93e800127cd4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f5157e00640 1 -- 172.21.15.28:0/3236895080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480b0070 con 0x7f51580af060 2024-03-21T08:41:31.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f5157e00640 1 -- 172.21.15.28:0/3236895080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51480ab600 con 0x7f51580af060 2024-03-21T08:41:31.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.681+0000 7f5157e00640 1 -- 172.21.15.28:0/3236895080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480ab9d0 con 0x7f51580af060 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- 172.21.15.28:0/3236895080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580af060 msgr2=0x7f51580cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580af060 0x7f51580cd380 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f51580cc140 tx=0x7f514801c1d0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- 172.21.15.28:0/3236895080 shutdown_connections 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- 172.21.15.28:0/3236895080 >> v2:172.21.15.151:3300/0 conn(0x7f51580b5350 0x7f5158071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b34f0 0x7f51580b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- 172.21.15.28:0/3236895080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580af060 0x7f51580cd380 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- 172.21.15.28:0/3236895080 >> 172.21.15.28:0/3236895080 conn(0x7f51580cc230 msgr2=0x7f51580cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- 172.21.15.28:0/3236895080 shutdown_connections 2024-03-21T08:41:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- 172.21.15.28:0/3236895080 wait complete. 2024-03-21T08:41:31.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 Processor -- start 2024-03-21T08:41:31.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- start start 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b5350 0x7f51581443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f515806d980 0x7f515806e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580b5730 con 0x7f51580b34f0 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515815c200 con 0x7f515806d980 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515fa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515815e630 con 0x7f51580b5350 2024-03-21T08:41:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55626/0 (socket says 172.21.15.28:55626) 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 -- 172.21.15.28:0/3223025121 learned_addr learned my addr 172.21.15.28:0/3223025121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515da00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f515806d980 0x7f515806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f514fe00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b5350 0x7f51581443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 -- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b5350 msgr2=0x7f51581443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b5350 0x7f51581443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 -- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f515806d980 msgr2=0x7f515806e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f515806d980 0x7f515806e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 -- 172.21.15.28:0/3223025121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515815e910 con 0x7f51580b34f0 2024-03-21T08:41:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f515d000640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f51480ae040 tx=0x7f514801aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.685+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480ba020 con 0x7f51580b34f0 2024-03-21T08:41:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51480b6420 con 0x7f51580b34f0 2024-03-21T08:41:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515815ebf0 con 0x7f51580b34f0 2024-03-21T08:41:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480b0040 con 0x7f51580b34f0 2024-03-21T08:41:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515815e320 con 0x7f51580b34f0 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515804f410 con 0x7f51580b34f0 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f51480bd020 con 0x7f51580b34f0 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f5155600640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 0x7f5134069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5148120270 con 0x7f51580b34f0 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f514fe00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 0x7f5134069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.689+0000 7f514fe00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 0x7f5134069fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5158151a20 tx=0x7f5140009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.693+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f51480c4030 con 0x7f51580b34f0 2024-03-21T08:41:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.853+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51580ccd50 con 0x7f5134067b20 2024-03-21T08:41:31.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f5155600640 1 -- 172.21.15.28:0/3223025121 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332538 (secure 0 0 0) 0x7f51580ccd50 con 0x7f5134067b20 2024-03-21T08:41:31.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 msgr2=0x7f5134069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 0x7f5134069fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5158151a20 tx=0x7f5140009190 comp rx=0 tx=0).stop 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 msgr2=0x7f515815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f51480ae040 tx=0x7f514801aa10 comp rx=0 tx=0).stop 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 shutdown_connections 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f515806d980 0x7f515806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5134067b20 0x7f5134069fe0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f51580b5350 0x7f51581443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 --2- 172.21.15.28:0/3223025121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51580b34f0 0x7f515815dde0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.865+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 >> 172.21.15.28:0/3223025121 conn(0x7f51580cc230 msgr2=0x7f51580ad690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.869+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 shutdown_connections 2024-03-21T08:41:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:31.869+0000 7f515fa00640 1 -- 172.21.15.28:0/3223025121 wait complete. 2024-03-21T08:41:31.891 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-21T08:41:31.724067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231892,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742828,"statfs":{"total":773094113280,"available":772856655872,"internally_reserved":0,"allocated":3317760,"data_stored":2007752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161451+0000","last_change":"2024-03-21T08:41:07.021438+0000","last_active":"2024-03-21T08:41:13.161451+0000","last_peered":"2024-03-21T08:41:13.161451+0000","last_clean":"2024-03-21T08:41:13.161451+0000","last_became_active":"2024-03-21T08:41:07.021142+0000","last_became_peered":"2024-03-21T08:41:07.021142+0000","last_unstale":"2024-03-21T08:41:13.161451+0000","last_undegraded":"2024-03-21T08:41:13.161451+0000","last_fullsized":"2024-03-21T08:41:13.161451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:15:32.936778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262630+0000","last_change":"2024-03-21T08:41:08.010424+0000","last_active":"2024-03-21T08:41:14.262630+0000","last_peered":"2024-03-21T08:41:14.262630+0000","last_clean":"2024-03-21T08:41:14.262630+0000","last_became_active":"2024-03-21T08:41:08.010169+0000","last_became_peered":"2024-03-21T08:41:08.010169+0000","last_unstale":"2024-03-21T08:41:14.262630+0000","last_undegraded":"2024-03-21T08:41:14.262630+0000","last_fullsized":"2024-03-21T08:41:14.262630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261510+0000","last_change":"2024-03-21T08:41:12.340891+0000","last_active":"2024-03-21T08:41:14.261510+0000","last_peered":"2024-03-21T08:41:14.261510+0000","last_clean":"2024-03-21T08:41:14.261510+0000","last_became_active":"2024-03-21T08:41:12.340181+0000","last_became_peered":"2024-03-21T08:41:12.340181+0000","last_unstale":"2024-03-21T08:41:14.261510+0000","last_undegraded":"2024-03-21T08:41:14.261510+0000","last_fullsized":"2024-03-21T08:41:14.261510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262757+0000","last_change":"2024-03-21T08:41:07.009640+0000","last_active":"2024-03-21T08:41:14.262757+0000","last_peered":"2024-03-21T08:41:14.262757+0000","last_clean":"2024-03-21T08:41:14.262757+0000","last_became_active":"2024-03-21T08:41:07.009390+0000","last_became_peered":"2024-03-21T08:41:07.009390+0000","last_unstale":"2024-03-21T08:41:14.262757+0000","last_undegraded":"2024-03-21T08:41:14.262757+0000","last_fullsized":"2024-03-21T08:41:14.262757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161250+0000","last_change":"2024-03-21T08:41:08.010008+0000","last_active":"2024-03-21T08:41:13.161250+0000","last_peered":"2024-03-21T08:41:13.161250+0000","last_clean":"2024-03-21T08:41:13.161250+0000","last_became_active":"2024-03-21T08:41:08.008163+0000","last_became_peered":"2024-03-21T08:41:08.008163+0000","last_unstale":"2024-03-21T08:41:13.161250+0000","last_undegraded":"2024-03-21T08:41:13.161250+0000","last_fullsized":"2024-03-21T08:41:13.161250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025449+0000","last_change":"2024-03-21T08:41:12.049570+0000","last_active":"2024-03-21T08:41:13.025449+0000","last_peered":"2024-03-21T08:41:13.025449+0000","last_clean":"2024-03-21T08:41:13.025449+0000","last_became_active":"2024-03-21T08:41:12.048903+0000","last_became_peered":"2024-03-21T08:41:12.048903+0000","last_unstale":"2024-03-21T08:41:13.025449+0000","last_undegraded":"2024-03-21T08:41:13.025449+0000","last_fullsized":"2024-03-21T08:41:13.025449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:10.935038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736993+0000","last_change":"2024-03-21T08:41:07.022988+0000","last_active":"2024-03-21T08:41:13.736993+0000","last_peered":"2024-03-21T08:41:13.736993+0000","last_clean":"2024-03-21T08:41:13.736993+0000","last_became_active":"2024-03-21T08:41:07.022643+0000","last_became_peered":"2024-03-21T08:41:07.022643+0000","last_unstale":"2024-03-21T08:41:13.736993+0000","last_undegraded":"2024-03-21T08:41:13.736993+0000","last_fullsized":"2024-03-21T08:41:13.736993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260935+0000","last_change":"2024-03-21T08:41:08.008250+0000","last_active":"2024-03-21T08:41:14.260935+0000","last_peered":"2024-03-21T08:41:14.260935+0000","last_clean":"2024-03-21T08:41:14.260935+0000","last_became_active":"2024-03-21T08:41:08.007638+0000","last_became_peered":"2024-03-21T08:41:08.007638+0000","last_unstale":"2024-03-21T08:41:14.260935+0000","last_undegraded":"2024-03-21T08:41:14.260935+0000","last_fullsized":"2024-03-21T08:41:14.260935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:14.189872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027230+0000","last_change":"2024-03-21T08:41:12.051136+0000","last_active":"2024-03-21T08:41:13.027230+0000","last_peered":"2024-03-21T08:41:13.027230+0000","last_clean":"2024-03-21T08:41:13.027230+0000","last_became_active":"2024-03-21T08:41:12.050706+0000","last_became_peered":"2024-03-21T08:41:12.050706+0000","last_unstale":"2024-03-21T08:41:13.027230+0000","last_undegraded":"2024-03-21T08:41:13.027230+0000","last_fullsized":"2024-03-21T08:41:13.027230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:28:03.320473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736865+0000","last_change":"2024-03-21T08:41:07.021316+0000","last_active":"2024-03-21T08:41:13.736865+0000","last_peered":"2024-03-21T08:41:13.736865+0000","last_clean":"2024-03-21T08:41:13.736865+0000","last_became_active":"2024-03-21T08:41:07.021039+0000","last_became_peered":"2024-03-21T08:41:07.021039+0000","last_unstale":"2024-03-21T08:41:13.736865+0000","last_undegraded":"2024-03-21T08:41:13.736865+0000","last_fullsized":"2024-03-21T08:41:13.736865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261060+0000","last_change":"2024-03-21T08:41:08.004273+0000","last_active":"2024-03-21T08:41:14.261060+0000","last_peered":"2024-03-21T08:41:14.261060+0000","last_clean":"2024-03-21T08:41:14.261060+0000","last_became_active":"2024-03-21T08:41:08.003800+0000","last_became_peered":"2024-03-21T08:41:08.003800+0000","last_unstale":"2024-03-21T08:41:14.261060+0000","last_undegraded":"2024-03-21T08:41:14.261060+0000","last_fullsized":"2024-03-21T08:41:14.261060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027771+0000","last_change":"2024-03-21T08:41:12.046634+0000","last_active":"2024-03-21T08:41:13.027771+0000","last_peered":"2024-03-21T08:41:13.027771+0000","last_clean":"2024-03-21T08:41:13.027771+0000","last_became_active":"2024-03-21T08:41:12.046224+0000","last_became_peered":"2024-03-21T08:41:12.046224+0000","last_unstale":"2024-03-21T08:41:13.027771+0000","last_undegraded":"2024-03-21T08:41:13.027771+0000","last_fullsized":"2024-03-21T08:41:13.027771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:20.757184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025520+0000","last_change":"2024-03-21T08:41:08.009427+0000","last_active":"2024-03-21T08:41:13.025520+0000","last_peered":"2024-03-21T08:41:13.025520+0000","last_clean":"2024-03-21T08:41:13.025520+0000","last_became_active":"2024-03-21T08:41:08.009082+0000","last_became_peered":"2024-03-21T08:41:08.009082+0000","last_unstale":"2024-03-21T08:41:13.025520+0000","last_undegraded":"2024-03-21T08:41:13.025520+0000","last_fullsized":"2024-03-21T08:41:13.025520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:07.174703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736774+0000","last_change":"2024-03-21T08:41:07.019968+0000","last_active":"2024-03-21T08:41:13.736774+0000","last_peered":"2024-03-21T08:41:13.736774+0000","last_clean":"2024-03-21T08:41:13.736774+0000","last_became_active":"2024-03-21T08:41:07.019039+0000","last_became_peered":"2024-03-21T08:41:07.019039+0000","last_unstale":"2024-03-21T08:41:13.736774+0000","last_undegraded":"2024-03-21T08:41:13.736774+0000","last_fullsized":"2024-03-21T08:41:13.736774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027696+0000","last_change":"2024-03-21T08:41:12.053357+0000","last_active":"2024-03-21T08:41:13.027696+0000","last_peered":"2024-03-21T08:41:13.027696+0000","last_clean":"2024-03-21T08:41:13.027696+0000","last_became_active":"2024-03-21T08:41:12.053045+0000","last_became_peered":"2024-03-21T08:41:12.053045+0000","last_unstale":"2024-03-21T08:41:13.027696+0000","last_undegraded":"2024-03-21T08:41:13.027696+0000","last_fullsized":"2024-03-21T08:41:13.027696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:52.060002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262174+0000","last_change":"2024-03-21T08:41:08.014032+0000","last_active":"2024-03-21T08:41:14.262174+0000","last_peered":"2024-03-21T08:41:14.262174+0000","last_clean":"2024-03-21T08:41:14.262174+0000","last_became_active":"2024-03-21T08:41:08.013716+0000","last_became_peered":"2024-03-21T08:41:08.013716+0000","last_unstale":"2024-03-21T08:41:14.262174+0000","last_undegraded":"2024-03-21T08:41:14.262174+0000","last_fullsized":"2024-03-21T08:41:14.262174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262296+0000","last_change":"2024-03-21T08:41:07.008347+0000","last_active":"2024-03-21T08:41:14.262296+0000","last_peered":"2024-03-21T08:41:14.262296+0000","last_clean":"2024-03-21T08:41:14.262296+0000","last_became_active":"2024-03-21T08:41:07.005441+0000","last_became_peered":"2024-03-21T08:41:07.005441+0000","last_unstale":"2024-03-21T08:41:14.262296+0000","last_undegraded":"2024-03-21T08:41:14.262296+0000","last_fullsized":"2024-03-21T08:41:14.262296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025374+0000","last_change":"2024-03-21T08:41:12.051147+0000","last_active":"2024-03-21T08:41:13.025374+0000","last_peered":"2024-03-21T08:41:13.025374+0000","last_clean":"2024-03-21T08:41:13.025374+0000","last_became_active":"2024-03-21T08:41:12.050744+0000","last_became_peered":"2024-03-21T08:41:12.050744+0000","last_unstale":"2024-03-21T08:41:13.025374+0000","last_undegraded":"2024-03-21T08:41:13.025374+0000","last_fullsized":"2024-03-21T08:41:13.025374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:15.657762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261625+0000","last_change":"2024-03-21T08:41:08.003261+0000","last_active":"2024-03-21T08:41:14.261625+0000","last_peered":"2024-03-21T08:41:14.261625+0000","last_clean":"2024-03-21T08:41:14.261625+0000","last_became_active":"2024-03-21T08:41:08.002934+0000","last_became_peered":"2024-03-21T08:41:08.002934+0000","last_unstale":"2024-03-21T08:41:14.261625+0000","last_undegraded":"2024-03-21T08:41:14.261625+0000","last_fullsized":"2024-03-21T08:41:14.261625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262016+0000","last_change":"2024-03-21T08:41:07.020788+0000","last_active":"2024-03-21T08:41:14.262016+0000","last_peered":"2024-03-21T08:41:14.262016+0000","last_clean":"2024-03-21T08:41:14.262016+0000","last_became_active":"2024-03-21T08:41:07.020341+0000","last_became_peered":"2024-03-21T08:41:07.020341+0000","last_unstale":"2024-03-21T08:41:14.262016+0000","last_undegraded":"2024-03-21T08:41:14.262016+0000","last_fullsized":"2024-03-21T08:41:14.262016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263514+0000","last_change":"2024-03-21T08:41:12.340482+0000","last_active":"2024-03-21T08:41:14.263514+0000","last_peered":"2024-03-21T08:41:14.263514+0000","last_clean":"2024-03-21T08:41:14.263514+0000","last_became_active":"2024-03-21T08:41:12.340076+0000","last_became_peered":"2024-03-21T08:41:12.340076+0000","last_unstale":"2024-03-21T08:41:14.263514+0000","last_undegraded":"2024-03-21T08:41:14.263514+0000","last_fullsized":"2024-03-21T08:41:14.263514+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:10.484672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027247+0000","last_change":"2024-03-21T08:41:08.002403+0000","last_active":"2024-03-21T08:41:13.027247+0000","last_peered":"2024-03-21T08:41:13.027247+0000","last_clean":"2024-03-21T08:41:13.027247+0000","last_became_active":"2024-03-21T08:41:08.001933+0000","last_became_peered":"2024-03-21T08:41:08.001933+0000","last_unstale":"2024-03-21T08:41:13.027247+0000","last_undegraded":"2024-03-21T08:41:13.027247+0000","last_fullsized":"2024-03-21T08:41:13.027247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:40.121158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736405+0000","last_change":"2024-03-21T08:41:07.019255+0000","last_active":"2024-03-21T08:41:13.736405+0000","last_peered":"2024-03-21T08:41:13.736405+0000","last_clean":"2024-03-21T08:41:13.736405+0000","last_became_active":"2024-03-21T08:41:07.018958+0000","last_became_peered":"2024-03-21T08:41:07.018958+0000","last_unstale":"2024-03-21T08:41:13.736405+0000","last_undegraded":"2024-03-21T08:41:13.736405+0000","last_fullsized":"2024-03-21T08:41:13.736405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027872+0000","last_change":"2024-03-21T08:41:12.049104+0000","last_active":"2024-03-21T08:41:13.027872+0000","last_peered":"2024-03-21T08:41:13.027872+0000","last_clean":"2024-03-21T08:41:13.027872+0000","last_became_active":"2024-03-21T08:41:12.048724+0000","last_became_peered":"2024-03-21T08:41:12.048724+0000","last_unstale":"2024-03-21T08:41:13.027872+0000","last_undegraded":"2024-03-21T08:41:13.027872+0000","last_fullsized":"2024-03-21T08:41:13.027872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161500+0000","last_change":"2024-03-21T08:41:08.006976+0000","last_active":"2024-03-21T08:41:13.161500+0000","last_peered":"2024-03-21T08:41:13.161500+0000","last_clean":"2024-03-21T08:41:13.161500+0000","last_became_active":"2024-03-21T08:41:08.006565+0000","last_became_peered":"2024-03-21T08:41:08.006565+0000","last_unstale":"2024-03-21T08:41:13.161500+0000","last_undegraded":"2024-03-21T08:41:13.161500+0000","last_fullsized":"2024-03-21T08:41:13.161500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026772+0000","last_change":"2024-03-21T08:41:07.014131+0000","last_active":"2024-03-21T08:41:13.026772+0000","last_peered":"2024-03-21T08:41:13.026772+0000","last_clean":"2024-03-21T08:41:13.026772+0000","last_became_active":"2024-03-21T08:41:07.013789+0000","last_became_peered":"2024-03-21T08:41:07.013789+0000","last_unstale":"2024-03-21T08:41:13.026772+0000","last_undegraded":"2024-03-21T08:41:13.026772+0000","last_fullsized":"2024-03-21T08:41:13.026772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027502+0000","last_change":"2024-03-21T08:41:12.336087+0000","last_active":"2024-03-21T08:41:13.027502+0000","last_peered":"2024-03-21T08:41:13.027502+0000","last_clean":"2024-03-21T08:41:13.027502+0000","last_became_active":"2024-03-21T08:41:12.335722+0000","last_became_peered":"2024-03-21T08:41:12.335722+0000","last_unstale":"2024-03-21T08:41:13.027502+0000","last_undegraded":"2024-03-21T08:41:13.027502+0000","last_fullsized":"2024-03-21T08:41:13.027502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:47.106105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262994+0000","last_change":"2024-03-21T08:41:08.021861+0000","last_active":"2024-03-21T08:41:14.262994+0000","last_peered":"2024-03-21T08:41:14.262994+0000","last_clean":"2024-03-21T08:41:14.262994+0000","last_became_active":"2024-03-21T08:41:08.021564+0000","last_became_peered":"2024-03-21T08:41:08.021564+0000","last_unstale":"2024-03-21T08:41:14.262994+0000","last_undegraded":"2024-03-21T08:41:14.262994+0000","last_fullsized":"2024-03-21T08:41:14.262994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263418+0000","last_change":"2024-03-21T08:41:07.010549+0000","last_active":"2024-03-21T08:41:14.263418+0000","last_peered":"2024-03-21T08:41:14.263418+0000","last_clean":"2024-03-21T08:41:14.263418+0000","last_became_active":"2024-03-21T08:41:07.010129+0000","last_became_peered":"2024-03-21T08:41:07.010129+0000","last_unstale":"2024-03-21T08:41:14.263418+0000","last_undegraded":"2024-03-21T08:41:14.263418+0000","last_fullsized":"2024-03-21T08:41:14.263418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261368+0000","last_change":"2024-03-21T08:41:12.340111+0000","last_active":"2024-03-21T08:41:14.261368+0000","last_peered":"2024-03-21T08:41:14.261368+0000","last_clean":"2024-03-21T08:41:14.261368+0000","last_became_active":"2024-03-21T08:41:12.339714+0000","last_became_peered":"2024-03-21T08:41:12.339714+0000","last_unstale":"2024-03-21T08:41:14.261368+0000","last_undegraded":"2024-03-21T08:41:14.261368+0000","last_fullsized":"2024-03-21T08:41:14.261368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:30:09.350945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262982+0000","last_change":"2024-03-21T08:41:08.005617+0000","last_active":"2024-03-21T08:41:14.262982+0000","last_peered":"2024-03-21T08:41:14.262982+0000","last_clean":"2024-03-21T08:41:14.262982+0000","last_became_active":"2024-03-21T08:41:08.005254+0000","last_became_peered":"2024-03-21T08:41:08.005254+0000","last_unstale":"2024-03-21T08:41:14.262982+0000","last_undegraded":"2024-03-21T08:41:14.262982+0000","last_fullsized":"2024-03-21T08:41:14.262982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737093+0000","last_change":"2024-03-21T08:41:07.019857+0000","last_active":"2024-03-21T08:41:13.737093+0000","last_peered":"2024-03-21T08:41:13.737093+0000","last_clean":"2024-03-21T08:41:13.737093+0000","last_became_active":"2024-03-21T08:41:07.019553+0000","last_became_peered":"2024-03-21T08:41:07.019553+0000","last_unstale":"2024-03-21T08:41:13.737093+0000","last_undegraded":"2024-03-21T08:41:13.737093+0000","last_fullsized":"2024-03-21T08:41:13.737093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:35:34.563261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161134+0000","last_change":"2024-03-21T08:41:12.335655+0000","last_active":"2024-03-21T08:41:13.161134+0000","last_peered":"2024-03-21T08:41:13.161134+0000","last_clean":"2024-03-21T08:41:13.161134+0000","last_became_active":"2024-03-21T08:41:12.335219+0000","last_became_peered":"2024-03-21T08:41:12.335219+0000","last_unstale":"2024-03-21T08:41:13.161134+0000","last_undegraded":"2024-03-21T08:41:13.161134+0000","last_fullsized":"2024-03-21T08:41:13.161134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261158+0000","last_change":"2024-03-21T08:41:08.016121+0000","last_active":"2024-03-21T08:41:14.261158+0000","last_peered":"2024-03-21T08:41:14.261158+0000","last_clean":"2024-03-21T08:41:14.261158+0000","last_became_active":"2024-03-21T08:41:08.015237+0000","last_became_peered":"2024-03-21T08:41:08.015237+0000","last_unstale":"2024-03-21T08:41:14.261158+0000","last_undegraded":"2024-03-21T08:41:14.261158+0000","last_fullsized":"2024-03-21T08:41:14.261158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:48.028184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263069+0000","last_change":"2024-03-21T08:41:07.013927+0000","last_active":"2024-03-21T08:41:14.263069+0000","last_peered":"2024-03-21T08:41:14.263069+0000","last_clean":"2024-03-21T08:41:14.263069+0000","last_became_active":"2024-03-21T08:41:07.013595+0000","last_became_peered":"2024-03-21T08:41:07.013595+0000","last_unstale":"2024-03-21T08:41:14.263069+0000","last_undegraded":"2024-03-21T08:41:14.263069+0000","last_fullsized":"2024-03-21T08:41:14.263069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262181+0000","last_change":"2024-03-21T08:41:12.056397+0000","last_active":"2024-03-21T08:41:14.262181+0000","last_peered":"2024-03-21T08:41:14.262181+0000","last_clean":"2024-03-21T08:41:14.262181+0000","last_became_active":"2024-03-21T08:41:12.054349+0000","last_became_peered":"2024-03-21T08:41:12.054349+0000","last_unstale":"2024-03-21T08:41:14.262181+0000","last_undegraded":"2024-03-21T08:41:14.262181+0000","last_fullsized":"2024-03-21T08:41:14.262181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027232+0000","last_change":"2024-03-21T08:41:07.997069+0000","last_active":"2024-03-21T08:41:13.027232+0000","last_peered":"2024-03-21T08:41:13.027232+0000","last_clean":"2024-03-21T08:41:13.027232+0000","last_became_active":"2024-03-21T08:41:07.996691+0000","last_became_peered":"2024-03-21T08:41:07.996691+0000","last_unstale":"2024-03-21T08:41:13.027232+0000","last_undegraded":"2024-03-21T08:41:13.027232+0000","last_fullsized":"2024-03-21T08:41:13.027232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:59.860172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260862+0000","last_change":"2024-03-21T08:41:07.004740+0000","last_active":"2024-03-21T08:41:14.260862+0000","last_peered":"2024-03-21T08:41:14.260862+0000","last_clean":"2024-03-21T08:41:14.260862+0000","last_became_active":"2024-03-21T08:41:07.004315+0000","last_became_peered":"2024-03-21T08:41:07.004315+0000","last_unstale":"2024-03-21T08:41:14.260862+0000","last_undegraded":"2024-03-21T08:41:14.260862+0000","last_fullsized":"2024-03-21T08:41:14.260862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:58.854457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263063+0000","last_change":"2024-03-21T08:41:12.049583+0000","last_active":"2024-03-21T08:41:14.263063+0000","last_peered":"2024-03-21T08:41:14.263063+0000","last_clean":"2024-03-21T08:41:14.263063+0000","last_became_active":"2024-03-21T08:41:12.048879+0000","last_became_peered":"2024-03-21T08:41:12.048879+0000","last_unstale":"2024-03-21T08:41:14.263063+0000","last_undegraded":"2024-03-21T08:41:14.263063+0000","last_fullsized":"2024-03-21T08:41:14.263063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:41.204023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261754+0000","last_change":"2024-03-21T08:41:08.006302+0000","last_active":"2024-03-21T08:41:14.261754+0000","last_peered":"2024-03-21T08:41:14.261754+0000","last_clean":"2024-03-21T08:41:14.261754+0000","last_became_active":"2024-03-21T08:41:08.005255+0000","last_became_peered":"2024-03-21T08:41:08.005255+0000","last_unstale":"2024-03-21T08:41:14.261754+0000","last_undegraded":"2024-03-21T08:41:14.261754+0000","last_fullsized":"2024-03-21T08:41:14.261754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025568+0000","last_change":"2024-03-21T08:41:07.004853+0000","last_active":"2024-03-21T08:41:13.025568+0000","last_peered":"2024-03-21T08:41:13.025568+0000","last_clean":"2024-03-21T08:41:13.025568+0000","last_became_active":"2024-03-21T08:41:07.004447+0000","last_became_peered":"2024-03-21T08:41:07.004447+0000","last_unstale":"2024-03-21T08:41:13.025568+0000","last_undegraded":"2024-03-21T08:41:13.025568+0000","last_fullsized":"2024-03-21T08:41:13.025568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263561+0000","last_change":"2024-03-21T08:41:12.057098+0000","last_active":"2024-03-21T08:41:14.263561+0000","last_peered":"2024-03-21T08:41:14.263561+0000","last_clean":"2024-03-21T08:41:14.263561+0000","last_became_active":"2024-03-21T08:41:12.056306+0000","last_became_peered":"2024-03-21T08:41:12.056306+0000","last_unstale":"2024-03-21T08:41:14.263561+0000","last_undegraded":"2024-03-21T08:41:14.263561+0000","last_fullsized":"2024-03-21T08:41:14.263561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:00:00.666748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160776+0000","last_change":"2024-03-21T08:41:07.995187+0000","last_active":"2024-03-21T08:41:13.160776+0000","last_peered":"2024-03-21T08:41:13.160776+0000","last_clean":"2024-03-21T08:41:13.160776+0000","last_became_active":"2024-03-21T08:41:07.994882+0000","last_became_peered":"2024-03-21T08:41:07.994882+0000","last_unstale":"2024-03-21T08:41:13.160776+0000","last_undegraded":"2024-03-21T08:41:13.160776+0000","last_fullsized":"2024-03-21T08:41:13.160776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:27.177399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736421+0000","last_change":"2024-03-21T08:41:07.019906+0000","last_active":"2024-03-21T08:41:13.736421+0000","last_peered":"2024-03-21T08:41:13.736421+0000","last_clean":"2024-03-21T08:41:13.736421+0000","last_became_active":"2024-03-21T08:41:07.019585+0000","last_became_peered":"2024-03-21T08:41:07.019585+0000","last_unstale":"2024-03-21T08:41:13.736421+0000","last_undegraded":"2024-03-21T08:41:13.736421+0000","last_fullsized":"2024-03-21T08:41:13.736421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161357+0000","last_change":"2024-03-21T08:41:12.337710+0000","last_active":"2024-03-21T08:41:13.161357+0000","last_peered":"2024-03-21T08:41:13.161357+0000","last_clean":"2024-03-21T08:41:13.161357+0000","last_became_active":"2024-03-21T08:41:12.336270+0000","last_became_peered":"2024-03-21T08:41:12.336270+0000","last_unstale":"2024-03-21T08:41:13.161357+0000","last_undegraded":"2024-03-21T08:41:13.161357+0000","last_fullsized":"2024-03-21T08:41:13.161357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:11.100710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261759+0000","last_change":"2024-03-21T08:41:08.015395+0000","last_active":"2024-03-21T08:41:14.261759+0000","last_peered":"2024-03-21T08:41:14.261759+0000","last_clean":"2024-03-21T08:41:14.261759+0000","last_became_active":"2024-03-21T08:41:08.015091+0000","last_became_peered":"2024-03-21T08:41:08.015091+0000","last_unstale":"2024-03-21T08:41:14.261759+0000","last_undegraded":"2024-03-21T08:41:14.261759+0000","last_fullsized":"2024-03-21T08:41:14.261759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260786+0000","last_change":"2024-03-21T08:41:07.008427+0000","last_active":"2024-03-21T08:41:14.260786+0000","last_peered":"2024-03-21T08:41:14.260786+0000","last_clean":"2024-03-21T08:41:14.260786+0000","last_became_active":"2024-03-21T08:41:07.008115+0000","last_became_peered":"2024-03-21T08:41:07.008115+0000","last_unstale":"2024-03-21T08:41:14.260786+0000","last_undegraded":"2024-03-21T08:41:14.260786+0000","last_fullsized":"2024-03-21T08:41:14.260786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736516+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:41:13.736516+0000","last_peered":"2024-03-21T08:41:13.736516+0000","last_clean":"2024-03-21T08:41:13.736516+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:41:13.736516+0000","last_undegraded":"2024-03-21T08:41:13.736516+0000","last_fullsized":"2024-03-21T08:41:13.736516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026534+0000","last_change":"2024-03-21T08:41:12.334882+0000","last_active":"2024-03-21T08:41:13.026534+0000","last_peered":"2024-03-21T08:41:13.026534+0000","last_clean":"2024-03-21T08:41:13.026534+0000","last_became_active":"2024-03-21T08:41:12.334459+0000","last_became_peered":"2024-03-21T08:41:12.334459+0000","last_unstale":"2024-03-21T08:41:13.026534+0000","last_undegraded":"2024-03-21T08:41:13.026534+0000","last_fullsized":"2024-03-21T08:41:13.026534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:17.097499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736898+0000","last_change":"2024-03-21T08:41:08.030635+0000","last_active":"2024-03-21T08:41:13.736898+0000","last_peered":"2024-03-21T08:41:13.736898+0000","last_clean":"2024-03-21T08:41:13.736898+0000","last_became_active":"2024-03-21T08:41:08.028635+0000","last_became_peered":"2024-03-21T08:41:08.028635+0000","last_unstale":"2024-03-21T08:41:13.736898+0000","last_undegraded":"2024-03-21T08:41:13.736898+0000","last_fullsized":"2024-03-21T08:41:13.736898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:14.013088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026041+0000","last_change":"2024-03-21T08:41:07.004839+0000","last_active":"2024-03-21T08:41:13.026041+0000","last_peered":"2024-03-21T08:41:13.026041+0000","last_clean":"2024-03-21T08:41:13.026041+0000","last_became_active":"2024-03-21T08:41:07.004409+0000","last_became_peered":"2024-03-21T08:41:07.004409+0000","last_unstale":"2024-03-21T08:41:13.026041+0000","last_undegraded":"2024-03-21T08:41:13.026041+0000","last_fullsized":"2024-03-21T08:41:13.026041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025597+0000","last_change":"2024-03-21T08:41:12.334998+0000","last_active":"2024-03-21T08:41:13.025597+0000","last_peered":"2024-03-21T08:41:13.025597+0000","last_clean":"2024-03-21T08:41:13.025597+0000","last_became_active":"2024-03-21T08:41:12.334610+0000","last_became_peered":"2024-03-21T08:41:12.334610+0000","last_unstale":"2024-03-21T08:41:13.025597+0000","last_undegraded":"2024-03-21T08:41:13.025597+0000","last_fullsized":"2024-03-21T08:41:13.025597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:04.540298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261002+0000","last_change":"2024-03-21T08:41:08.014545+0000","last_active":"2024-03-21T08:41:14.261002+0000","last_peered":"2024-03-21T08:41:14.261002+0000","last_clean":"2024-03-21T08:41:14.261002+0000","last_became_active":"2024-03-21T08:41:08.013815+0000","last_became_peered":"2024-03-21T08:41:08.013815+0000","last_unstale":"2024-03-21T08:41:14.261002+0000","last_undegraded":"2024-03-21T08:41:14.261002+0000","last_fullsized":"2024-03-21T08:41:14.261002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:07.068892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263256+0000","last_change":"2024-03-21T08:41:07.011021+0000","last_active":"2024-03-21T08:41:14.263256+0000","last_peered":"2024-03-21T08:41:14.263256+0000","last_clean":"2024-03-21T08:41:14.263256+0000","last_became_active":"2024-03-21T08:41:07.010411+0000","last_became_peered":"2024-03-21T08:41:07.010411+0000","last_unstale":"2024-03-21T08:41:14.263256+0000","last_undegraded":"2024-03-21T08:41:14.263256+0000","last_fullsized":"2024-03-21T08:41:14.263256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161403+0000","last_change":"2024-03-21T08:41:12.055570+0000","last_active":"2024-03-21T08:41:13.161403+0000","last_peered":"2024-03-21T08:41:13.161403+0000","last_clean":"2024-03-21T08:41:13.161403+0000","last_became_active":"2024-03-21T08:41:12.055174+0000","last_became_peered":"2024-03-21T08:41:12.055174+0000","last_unstale":"2024-03-21T08:41:13.161403+0000","last_undegraded":"2024-03-21T08:41:13.161403+0000","last_fullsized":"2024-03-21T08:41:13.161403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:45:15.642543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737253+0000","last_change":"2024-03-21T08:41:08.037018+0000","last_active":"2024-03-21T08:41:13.737253+0000","last_peered":"2024-03-21T08:41:13.737253+0000","last_clean":"2024-03-21T08:41:13.737253+0000","last_became_active":"2024-03-21T08:41:08.036676+0000","last_became_peered":"2024-03-21T08:41:08.036676+0000","last_unstale":"2024-03-21T08:41:13.737253+0000","last_undegraded":"2024-03-21T08:41:13.737253+0000","last_fullsized":"2024-03-21T08:41:13.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:59:56.634930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025975+0000","last_change":"2024-03-21T08:41:07.004711+0000","last_active":"2024-03-21T08:41:13.025975+0000","last_peered":"2024-03-21T08:41:13.025975+0000","last_clean":"2024-03-21T08:41:13.025975+0000","last_became_active":"2024-03-21T08:41:07.004192+0000","last_became_peered":"2024-03-21T08:41:07.004192+0000","last_unstale":"2024-03-21T08:41:13.025975+0000","last_undegraded":"2024-03-21T08:41:13.025975+0000","last_fullsized":"2024-03-21T08:41:13.025975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260477+0000","last_change":"2024-03-21T08:41:12.341167+0000","last_active":"2024-03-21T08:41:14.260477+0000","last_peered":"2024-03-21T08:41:14.260477+0000","last_clean":"2024-03-21T08:41:14.260477+0000","last_became_active":"2024-03-21T08:41:12.339602+0000","last_became_peered":"2024-03-21T08:41:12.339602+0000","last_unstale":"2024-03-21T08:41:14.260477+0000","last_undegraded":"2024-03-21T08:41:14.260477+0000","last_fullsized":"2024-03-21T08:41:14.260477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737088+0000","last_change":"2024-03-21T08:41:08.033751+0000","last_active":"2024-03-21T08:41:13.737088+0000","last_peered":"2024-03-21T08:41:13.737088+0000","last_clean":"2024-03-21T08:41:13.737088+0000","last_became_active":"2024-03-21T08:41:08.033458+0000","last_became_peered":"2024-03-21T08:41:08.033458+0000","last_unstale":"2024-03-21T08:41:13.737088+0000","last_undegraded":"2024-03-21T08:41:13.737088+0000","last_fullsized":"2024-03-21T08:41:13.737088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:13.065681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027669+0000","last_change":"2024-03-21T08:41:07.017025+0000","last_active":"2024-03-21T08:41:13.027669+0000","last_peered":"2024-03-21T08:41:13.027669+0000","last_clean":"2024-03-21T08:41:13.027669+0000","last_became_active":"2024-03-21T08:41:07.016654+0000","last_became_peered":"2024-03-21T08:41:07.016654+0000","last_unstale":"2024-03-21T08:41:13.027669+0000","last_undegraded":"2024-03-21T08:41:13.027669+0000","last_fullsized":"2024-03-21T08:41:13.027669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:10:03.436446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025227+0000","last_change":"2024-03-21T08:41:12.048497+0000","last_active":"2024-03-21T08:41:13.025227+0000","last_peered":"2024-03-21T08:41:13.025227+0000","last_clean":"2024-03-21T08:41:13.025227+0000","last_became_active":"2024-03-21T08:41:12.047987+0000","last_became_peered":"2024-03-21T08:41:12.047987+0000","last_unstale":"2024-03-21T08:41:13.025227+0000","last_undegraded":"2024-03-21T08:41:13.025227+0000","last_fullsized":"2024-03-21T08:41:13.025227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:39.659897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736537+0000","last_change":"2024-03-21T08:41:08.033999+0000","last_active":"2024-03-21T08:41:13.736537+0000","last_peered":"2024-03-21T08:41:13.736537+0000","last_clean":"2024-03-21T08:41:13.736537+0000","last_became_active":"2024-03-21T08:41:08.033015+0000","last_became_peered":"2024-03-21T08:41:08.033015+0000","last_unstale":"2024-03-21T08:41:13.736537+0000","last_undegraded":"2024-03-21T08:41:13.736537+0000","last_fullsized":"2024-03-21T08:41:13.736537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:16:25.591202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025409+0000","last_change":"2024-03-21T08:41:07.001332+0000","last_active":"2024-03-21T08:41:13.025409+0000","last_peered":"2024-03-21T08:41:13.025409+0000","last_clean":"2024-03-21T08:41:13.025409+0000","last_became_active":"2024-03-21T08:41:06.999605+0000","last_became_peered":"2024-03-21T08:41:06.999605+0000","last_unstale":"2024-03-21T08:41:13.025409+0000","last_undegraded":"2024-03-21T08:41:13.025409+0000","last_fullsized":"2024-03-21T08:41:13.025409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:06:31.096538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161359+0000","last_change":"2024-03-21T08:41:12.059105+0000","last_active":"2024-03-21T08:41:13.161359+0000","last_peered":"2024-03-21T08:41:13.161359+0000","last_clean":"2024-03-21T08:41:13.161359+0000","last_became_active":"2024-03-21T08:41:12.058643+0000","last_became_peered":"2024-03-21T08:41:12.058643+0000","last_unstale":"2024-03-21T08:41:13.161359+0000","last_undegraded":"2024-03-21T08:41:13.161359+0000","last_fullsized":"2024-03-21T08:41:13.161359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.221689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025534+0000","last_change":"2024-03-21T08:41:08.010609+0000","last_active":"2024-03-21T08:41:13.025534+0000","last_peered":"2024-03-21T08:41:13.025534+0000","last_clean":"2024-03-21T08:41:13.025534+0000","last_became_active":"2024-03-21T08:41:08.010109+0000","last_became_peered":"2024-03-21T08:41:08.010109+0000","last_unstale":"2024-03-21T08:41:13.025534+0000","last_undegraded":"2024-03-21T08:41:13.025534+0000","last_fullsized":"2024-03-21T08:41:13.025534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026017+0000","last_change":"2024-03-21T08:41:07.010384+0000","last_active":"2024-03-21T08:41:13.026017+0000","last_peered":"2024-03-21T08:41:13.026017+0000","last_clean":"2024-03-21T08:41:13.026017+0000","last_became_active":"2024-03-21T08:41:07.010019+0000","last_became_peered":"2024-03-21T08:41:07.010019+0000","last_unstale":"2024-03-21T08:41:13.026017+0000","last_undegraded":"2024-03-21T08:41:13.026017+0000","last_fullsized":"2024-03-21T08:41:13.026017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262930+0000","last_change":"2024-03-21T08:41:12.341603+0000","last_active":"2024-03-21T08:41:14.262930+0000","last_peered":"2024-03-21T08:41:14.262930+0000","last_clean":"2024-03-21T08:41:14.262930+0000","last_became_active":"2024-03-21T08:41:12.341158+0000","last_became_peered":"2024-03-21T08:41:12.341158+0000","last_unstale":"2024-03-21T08:41:14.262930+0000","last_undegraded":"2024-03-21T08:41:14.262930+0000","last_fullsized":"2024-03-21T08:41:14.262930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:02.246810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736756+0000","last_change":"2024-03-21T08:41:08.032505+0000","last_active":"2024-03-21T08:41:13.736756+0000","last_peered":"2024-03-21T08:41:13.736756+0000","last_clean":"2024-03-21T08:41:13.736756+0000","last_became_active":"2024-03-21T08:41:08.032026+0000","last_became_peered":"2024-03-21T08:41:08.032026+0000","last_unstale":"2024-03-21T08:41:13.736756+0000","last_undegraded":"2024-03-21T08:41:13.736756+0000","last_fullsized":"2024-03-21T08:41:13.736756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:21.306665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260844+0000","last_change":"2024-03-21T08:41:07.003264+0000","last_active":"2024-03-21T08:41:14.260844+0000","last_peered":"2024-03-21T08:41:14.260844+0000","last_clean":"2024-03-21T08:41:14.260844+0000","last_became_active":"2024-03-21T08:41:07.002969+0000","last_became_peered":"2024-03-21T08:41:07.002969+0000","last_unstale":"2024-03-21T08:41:14.260844+0000","last_undegraded":"2024-03-21T08:41:14.260844+0000","last_fullsized":"2024-03-21T08:41:14.260844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:19.390331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262423+0000","last_change":"2024-03-21T08:41:12.336949+0000","last_active":"2024-03-21T08:41:14.262423+0000","last_peered":"2024-03-21T08:41:14.262423+0000","last_clean":"2024-03-21T08:41:14.262423+0000","last_became_active":"2024-03-21T08:41:12.335654+0000","last_became_peered":"2024-03-21T08:41:12.335654+0000","last_unstale":"2024-03-21T08:41:14.262423+0000","last_undegraded":"2024-03-21T08:41:14.262423+0000","last_fullsized":"2024-03-21T08:41:14.262423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:54:40.677358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160969+0000","last_change":"2024-03-21T08:41:08.020852+0000","last_active":"2024-03-21T08:41:13.160969+0000","last_peered":"2024-03-21T08:41:13.160969+0000","last_clean":"2024-03-21T08:41:13.160969+0000","last_became_active":"2024-03-21T08:41:08.020547+0000","last_became_peered":"2024-03-21T08:41:08.020547+0000","last_unstale":"2024-03-21T08:41:13.160969+0000","last_undegraded":"2024-03-21T08:41:13.160969+0000","last_fullsized":"2024-03-21T08:41:13.160969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:56.518303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161459+0000","last_change":"2024-03-21T08:41:07.013979+0000","last_active":"2024-03-21T08:41:13.161459+0000","last_peered":"2024-03-21T08:41:13.161459+0000","last_clean":"2024-03-21T08:41:13.161459+0000","last_became_active":"2024-03-21T08:41:07.013725+0000","last_became_peered":"2024-03-21T08:41:07.013725+0000","last_unstale":"2024-03-21T08:41:13.161459+0000","last_undegraded":"2024-03-21T08:41:13.161459+0000","last_fullsized":"2024-03-21T08:41:13.161459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:03:13.978443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261111+0000","last_change":"2024-03-21T08:41:12.335008+0000","last_active":"2024-03-21T08:41:14.261111+0000","last_peered":"2024-03-21T08:41:14.261111+0000","last_clean":"2024-03-21T08:41:14.261111+0000","last_became_active":"2024-03-21T08:41:12.334635+0000","last_became_peered":"2024-03-21T08:41:12.334635+0000","last_unstale":"2024-03-21T08:41:14.261111+0000","last_undegraded":"2024-03-21T08:41:14.261111+0000","last_fullsized":"2024-03-21T08:41:14.261111+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736888+0000","last_change":"2024-03-21T08:41:08.026214+0000","last_active":"2024-03-21T08:41:13.736888+0000","last_peered":"2024-03-21T08:41:13.736888+0000","last_clean":"2024-03-21T08:41:13.736888+0000","last_became_active":"2024-03-21T08:41:08.025686+0000","last_became_peered":"2024-03-21T08:41:08.025686+0000","last_unstale":"2024-03-21T08:41:13.736888+0000","last_undegraded":"2024-03-21T08:41:13.736888+0000","last_fullsized":"2024-03-21T08:41:13.736888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:32:26.184390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:23.028503+0000","last_change":"2024-03-21T08:41:07.005099+0000","last_active":"2024-03-21T08:41:23.028503+0000","last_peered":"2024-03-21T08:41:23.028503+0000","last_clean":"2024-03-21T08:41:23.028503+0000","last_became_active":"2024-03-21T08:41:07.004825+0000","last_became_peered":"2024-03-21T08:41:07.004825+0000","last_unstale":"2024-03-21T08:41:23.028503+0000","last_undegraded":"2024-03-21T08:41:23.028503+0000","last_fullsized":"2024-03-21T08:41:23.028503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262224+0000","last_change":"2024-03-21T08:41:12.057213+0000","last_active":"2024-03-21T08:41:14.262224+0000","last_peered":"2024-03-21T08:41:14.262224+0000","last_clean":"2024-03-21T08:41:14.262224+0000","last_became_active":"2024-03-21T08:41:12.056539+0000","last_became_peered":"2024-03-21T08:41:12.056539+0000","last_unstale":"2024-03-21T08:41:14.262224+0000","last_undegraded":"2024-03-21T08:41:14.262224+0000","last_fullsized":"2024-03-21T08:41:14.262224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:35.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027129+0000","last_change":"2024-03-21T08:41:08.018576+0000","last_active":"2024-03-21T08:41:13.027129+0000","last_peered":"2024-03-21T08:41:13.027129+0000","last_clean":"2024-03-21T08:41:13.027129+0000","last_became_active":"2024-03-21T08:41:08.018277+0000","last_became_peered":"2024-03-21T08:41:08.018277+0000","last_unstale":"2024-03-21T08:41:13.027129+0000","last_undegraded":"2024-03-21T08:41:13.027129+0000","last_fullsized":"2024-03-21T08:41:13.027129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:00:26.730038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263165+0000","last_change":"2024-03-21T08:41:07.013500+0000","last_active":"2024-03-21T08:41:14.263165+0000","last_peered":"2024-03-21T08:41:14.263165+0000","last_clean":"2024-03-21T08:41:14.263165+0000","last_became_active":"2024-03-21T08:41:07.010794+0000","last_became_peered":"2024-03-21T08:41:07.010794+0000","last_unstale":"2024-03-21T08:41:14.263165+0000","last_undegraded":"2024-03-21T08:41:14.263165+0000","last_fullsized":"2024-03-21T08:41:14.263165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027024+0000","last_change":"2024-03-21T08:41:12.049156+0000","last_active":"2024-03-21T08:41:13.027024+0000","last_peered":"2024-03-21T08:41:13.027024+0000","last_clean":"2024-03-21T08:41:13.027024+0000","last_became_active":"2024-03-21T08:41:12.048737+0000","last_became_peered":"2024-03-21T08:41:12.048737+0000","last_unstale":"2024-03-21T08:41:13.027024+0000","last_undegraded":"2024-03-21T08:41:13.027024+0000","last_fullsized":"2024-03-21T08:41:13.027024+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:12.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161355+0000","last_change":"2024-03-21T08:41:08.012883+0000","last_active":"2024-03-21T08:41:13.161355+0000","last_peered":"2024-03-21T08:41:13.161355+0000","last_clean":"2024-03-21T08:41:13.161355+0000","last_became_active":"2024-03-21T08:41:08.012565+0000","last_became_peered":"2024-03-21T08:41:08.012565+0000","last_unstale":"2024-03-21T08:41:13.161355+0000","last_undegraded":"2024-03-21T08:41:13.161355+0000","last_fullsized":"2024-03-21T08:41:13.161355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261096+0000","last_change":"2024-03-21T08:41:07.013115+0000","last_active":"2024-03-21T08:41:14.261096+0000","last_peered":"2024-03-21T08:41:14.261096+0000","last_clean":"2024-03-21T08:41:14.261096+0000","last_became_active":"2024-03-21T08:41:07.012884+0000","last_became_peered":"2024-03-21T08:41:07.012884+0000","last_unstale":"2024-03-21T08:41:14.261096+0000","last_undegraded":"2024-03-21T08:41:14.261096+0000","last_fullsized":"2024-03-21T08:41:14.261096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:39.115443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261540+0000","last_change":"2024-03-21T08:41:12.051266+0000","last_active":"2024-03-21T08:41:14.261540+0000","last_peered":"2024-03-21T08:41:14.261540+0000","last_clean":"2024-03-21T08:41:14.261540+0000","last_became_active":"2024-03-21T08:41:12.050452+0000","last_became_peered":"2024-03-21T08:41:12.050452+0000","last_unstale":"2024-03-21T08:41:14.261540+0000","last_undegraded":"2024-03-21T08:41:14.261540+0000","last_fullsized":"2024-03-21T08:41:14.261540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261053+0000","last_change":"2024-03-21T08:41:08.029184+0000","last_active":"2024-03-21T08:41:14.261053+0000","last_peered":"2024-03-21T08:41:14.261053+0000","last_clean":"2024-03-21T08:41:14.261053+0000","last_became_active":"2024-03-21T08:41:08.028897+0000","last_became_peered":"2024-03-21T08:41:08.028897+0000","last_unstale":"2024-03-21T08:41:14.261053+0000","last_undegraded":"2024-03-21T08:41:14.261053+0000","last_fullsized":"2024-03-21T08:41:14.261053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:59.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026681+0000","last_change":"2024-03-21T08:41:07.021159+0000","last_active":"2024-03-21T08:41:13.026681+0000","last_peered":"2024-03-21T08:41:13.026681+0000","last_clean":"2024-03-21T08:41:13.026681+0000","last_became_active":"2024-03-21T08:41:07.020879+0000","last_became_peered":"2024-03-21T08:41:07.020879+0000","last_unstale":"2024-03-21T08:41:13.026681+0000","last_undegraded":"2024-03-21T08:41:13.026681+0000","last_fullsized":"2024-03-21T08:41:13.026681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261676+0000","last_change":"2024-03-21T08:41:12.340577+0000","last_active":"2024-03-21T08:41:14.261676+0000","last_peered":"2024-03-21T08:41:14.261676+0000","last_clean":"2024-03-21T08:41:14.261676+0000","last_became_active":"2024-03-21T08:41:12.339629+0000","last_became_peered":"2024-03-21T08:41:12.339629+0000","last_unstale":"2024-03-21T08:41:14.261676+0000","last_undegraded":"2024-03-21T08:41:14.261676+0000","last_fullsized":"2024-03-21T08:41:14.261676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:42.494428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161192+0000","last_change":"2024-03-21T08:41:12.340677+0000","last_active":"2024-03-21T08:41:13.161192+0000","last_peered":"2024-03-21T08:41:13.161192+0000","last_clean":"2024-03-21T08:41:13.161192+0000","last_became_active":"2024-03-21T08:41:12.340153+0000","last_became_peered":"2024-03-21T08:41:12.340153+0000","last_unstale":"2024-03-21T08:41:13.161192+0000","last_undegraded":"2024-03-21T08:41:13.161192+0000","last_fullsized":"2024-03-21T08:41:13.161192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:48.677726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261917+0000","last_change":"2024-03-21T08:41:08.004430+0000","last_active":"2024-03-21T08:41:14.261917+0000","last_peered":"2024-03-21T08:41:14.261917+0000","last_clean":"2024-03-21T08:41:14.261917+0000","last_became_active":"2024-03-21T08:41:08.003935+0000","last_became_peered":"2024-03-21T08:41:08.003935+0000","last_unstale":"2024-03-21T08:41:14.261917+0000","last_undegraded":"2024-03-21T08:41:14.261917+0000","last_fullsized":"2024-03-21T08:41:14.261917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260592+0000","last_change":"2024-03-21T08:41:07.011532+0000","last_active":"2024-03-21T08:41:14.260592+0000","last_peered":"2024-03-21T08:41:14.260592+0000","last_clean":"2024-03-21T08:41:14.260592+0000","last_became_active":"2024-03-21T08:41:07.008460+0000","last_became_peered":"2024-03-21T08:41:07.008460+0000","last_unstale":"2024-03-21T08:41:14.260592+0000","last_undegraded":"2024-03-21T08:41:14.260592+0000","last_fullsized":"2024-03-21T08:41:14.260592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026625+0000","last_change":"2024-03-21T08:41:12.054672+0000","last_active":"2024-03-21T08:41:13.026625+0000","last_peered":"2024-03-21T08:41:13.026625+0000","last_clean":"2024-03-21T08:41:13.026625+0000","last_became_active":"2024-03-21T08:41:12.054257+0000","last_became_peered":"2024-03-21T08:41:12.054257+0000","last_unstale":"2024-03-21T08:41:13.026625+0000","last_undegraded":"2024-03-21T08:41:13.026625+0000","last_fullsized":"2024-03-21T08:41:13.026625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261534+0000","last_change":"2024-03-21T08:41:08.015362+0000","last_active":"2024-03-21T08:41:14.261534+0000","last_peered":"2024-03-21T08:41:14.261534+0000","last_clean":"2024-03-21T08:41:14.261534+0000","last_became_active":"2024-03-21T08:41:08.015110+0000","last_became_peered":"2024-03-21T08:41:08.015110+0000","last_unstale":"2024-03-21T08:41:14.261534+0000","last_undegraded":"2024-03-21T08:41:14.261534+0000","last_fullsized":"2024-03-21T08:41:14.261534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261823+0000","last_change":"2024-03-21T08:41:07.001659+0000","last_active":"2024-03-21T08:41:14.261823+0000","last_peered":"2024-03-21T08:41:14.261823+0000","last_clean":"2024-03-21T08:41:14.261823+0000","last_became_active":"2024-03-21T08:41:07.001354+0000","last_became_peered":"2024-03-21T08:41:07.001354+0000","last_unstale":"2024-03-21T08:41:14.261823+0000","last_undegraded":"2024-03-21T08:41:14.261823+0000","last_fullsized":"2024-03-21T08:41:14.261823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.256597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026173+0000","last_change":"2024-03-21T08:41:07.009459+0000","last_active":"2024-03-21T08:41:13.026173+0000","last_peered":"2024-03-21T08:41:13.026173+0000","last_clean":"2024-03-21T08:41:13.026173+0000","last_became_active":"2024-03-21T08:41:07.009113+0000","last_became_peered":"2024-03-21T08:41:07.009113+0000","last_unstale":"2024-03-21T08:41:13.026173+0000","last_undegraded":"2024-03-21T08:41:13.026173+0000","last_fullsized":"2024-03-21T08:41:13.026173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161366+0000","last_change":"2024-03-21T08:41:07.996784+0000","last_active":"2024-03-21T08:41:13.161366+0000","last_peered":"2024-03-21T08:41:13.161366+0000","last_clean":"2024-03-21T08:41:13.161366+0000","last_became_active":"2024-03-21T08:41:07.996521+0000","last_became_peered":"2024-03-21T08:41:07.996521+0000","last_unstale":"2024-03-21T08:41:13.161366+0000","last_undegraded":"2024-03-21T08:41:13.161366+0000","last_fullsized":"2024-03-21T08:41:13.161366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025773+0000","last_change":"2024-03-21T08:41:12.052094+0000","last_active":"2024-03-21T08:41:13.025773+0000","last_peered":"2024-03-21T08:41:13.025773+0000","last_clean":"2024-03-21T08:41:13.025773+0000","last_became_active":"2024-03-21T08:41:12.051727+0000","last_became_peered":"2024-03-21T08:41:12.051727+0000","last_unstale":"2024-03-21T08:41:13.025773+0000","last_undegraded":"2024-03-21T08:41:13.025773+0000","last_fullsized":"2024-03-21T08:41:13.025773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:06:49.718295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262269+0000","last_change":"2024-03-21T08:41:07.012156+0000","last_active":"2024-03-21T08:41:14.262269+0000","last_peered":"2024-03-21T08:41:14.262269+0000","last_clean":"2024-03-21T08:41:14.262269+0000","last_became_active":"2024-03-21T08:41:07.011881+0000","last_became_peered":"2024-03-21T08:41:07.011881+0000","last_unstale":"2024-03-21T08:41:14.262269+0000","last_undegraded":"2024-03-21T08:41:14.262269+0000","last_fullsized":"2024-03-21T08:41:14.262269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027548+0000","last_change":"2024-03-21T08:41:08.002324+0000","last_active":"2024-03-21T08:41:13.027548+0000","last_peered":"2024-03-21T08:41:13.027548+0000","last_clean":"2024-03-21T08:41:13.027548+0000","last_became_active":"2024-03-21T08:41:08.002038+0000","last_became_peered":"2024-03-21T08:41:08.002038+0000","last_unstale":"2024-03-21T08:41:13.027548+0000","last_undegraded":"2024-03-21T08:41:13.027548+0000","last_fullsized":"2024-03-21T08:41:13.027548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:20.396047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262336+0000","last_change":"2024-03-21T08:41:12.058878+0000","last_active":"2024-03-21T08:41:14.262336+0000","last_peered":"2024-03-21T08:41:14.262336+0000","last_clean":"2024-03-21T08:41:14.262336+0000","last_became_active":"2024-03-21T08:41:12.058446+0000","last_became_peered":"2024-03-21T08:41:12.058446+0000","last_unstale":"2024-03-21T08:41:14.262336+0000","last_undegraded":"2024-03-21T08:41:14.262336+0000","last_fullsized":"2024-03-21T08:41:14.262336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:59:29.179384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025278+0000","last_change":"2024-03-21T08:41:08.010406+0000","last_active":"2024-03-21T08:41:13.025278+0000","last_peered":"2024-03-21T08:41:13.025278+0000","last_clean":"2024-03-21T08:41:13.025278+0000","last_became_active":"2024-03-21T08:41:08.009763+0000","last_became_peered":"2024-03-21T08:41:08.009763+0000","last_unstale":"2024-03-21T08:41:13.025278+0000","last_undegraded":"2024-03-21T08:41:13.025278+0000","last_fullsized":"2024-03-21T08:41:13.025278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025559+0000","last_change":"2024-03-21T08:41:08.009958+0000","last_active":"2024-03-21T08:41:13.025559+0000","last_peered":"2024-03-21T08:41:13.025559+0000","last_clean":"2024-03-21T08:41:13.025559+0000","last_became_active":"2024-03-21T08:41:08.009691+0000","last_became_peered":"2024-03-21T08:41:08.009691+0000","last_unstale":"2024-03-21T08:41:13.025559+0000","last_undegraded":"2024-03-21T08:41:13.025559+0000","last_fullsized":"2024-03-21T08:41:13.025559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:46.141146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260744+0000","last_change":"2024-03-21T08:41:08.004794+0000","last_active":"2024-03-21T08:41:14.260744+0000","last_peered":"2024-03-21T08:41:14.260744+0000","last_clean":"2024-03-21T08:41:14.260744+0000","last_became_active":"2024-03-21T08:41:08.003470+0000","last_became_peered":"2024-03-21T08:41:08.003470+0000","last_unstale":"2024-03-21T08:41:14.260744+0000","last_undegraded":"2024-03-21T08:41:14.260744+0000","last_fullsized":"2024-03-21T08:41:14.260744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161263+0000","last_change":"2024-03-21T08:41:08.014560+0000","last_active":"2024-03-21T08:41:13.161263+0000","last_peered":"2024-03-21T08:41:13.161263+0000","last_clean":"2024-03-21T08:41:13.161263+0000","last_became_active":"2024-03-21T08:41:08.014307+0000","last_became_peered":"2024-03-21T08:41:08.014307+0000","last_unstale":"2024-03-21T08:41:13.161263+0000","last_undegraded":"2024-03-21T08:41:13.161263+0000","last_fullsized":"2024-03-21T08:41:13.161263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262543+0000","last_change":"2024-03-21T08:41:08.026095+0000","last_active":"2024-03-21T08:41:14.262543+0000","last_peered":"2024-03-21T08:41:14.262543+0000","last_clean":"2024-03-21T08:41:14.262543+0000","last_became_active":"2024-03-21T08:41:08.025719+0000","last_became_peered":"2024-03-21T08:41:08.025719+0000","last_unstale":"2024-03-21T08:41:14.262543+0000","last_undegraded":"2024-03-21T08:41:14.262543+0000","last_fullsized":"2024-03-21T08:41:14.262543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262685+0000","last_change":"2024-03-21T08:41:08.007171+0000","last_active":"2024-03-21T08:41:14.262685+0000","last_peered":"2024-03-21T08:41:14.262685+0000","last_clean":"2024-03-21T08:41:14.262685+0000","last_became_active":"2024-03-21T08:41:08.006702+0000","last_became_peered":"2024-03-21T08:41:08.006702+0000","last_unstale":"2024-03-21T08:41:14.262685+0000","last_undegraded":"2024-03-21T08:41:14.262685+0000","last_fullsized":"2024-03-21T08:41:14.262685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026846+0000","last_change":"2024-03-21T08:41:08.029843+0000","last_active":"2024-03-21T08:41:13.026846+0000","last_peered":"2024-03-21T08:41:13.026846+0000","last_clean":"2024-03-21T08:41:13.026846+0000","last_became_active":"2024-03-21T08:41:08.029528+0000","last_became_peered":"2024-03-21T08:41:08.029528+0000","last_unstale":"2024-03-21T08:41:13.026846+0000","last_undegraded":"2024-03-21T08:41:13.026846+0000","last_fullsized":"2024-03-21T08:41:13.026846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262888+0000","last_change":"2024-03-21T08:41:08.013879+0000","last_active":"2024-03-21T08:41:14.262888+0000","last_peered":"2024-03-21T08:41:14.262888+0000","last_clean":"2024-03-21T08:41:14.262888+0000","last_became_active":"2024-03-21T08:41:08.013610+0000","last_became_peered":"2024-03-21T08:41:08.013610+0000","last_unstale":"2024-03-21T08:41:14.262888+0000","last_undegraded":"2024-03-21T08:41:14.262888+0000","last_fullsized":"2024-03-21T08:41:14.262888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260551+0000","last_change":"2024-03-21T08:41:08.001171+0000","last_active":"2024-03-21T08:41:14.260551+0000","last_peered":"2024-03-21T08:41:14.260551+0000","last_clean":"2024-03-21T08:41:14.260551+0000","last_became_active":"2024-03-21T08:41:08.000833+0000","last_became_peered":"2024-03-21T08:41:08.000833+0000","last_unstale":"2024-03-21T08:41:14.260551+0000","last_undegraded":"2024-03-21T08:41:14.260551+0000","last_fullsized":"2024-03-21T08:41:14.260551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261730+0000","last_change":"2024-03-21T08:41:08.014405+0000","last_active":"2024-03-21T08:41:14.261730+0000","last_peered":"2024-03-21T08:41:14.261730+0000","last_clean":"2024-03-21T08:41:14.261730+0000","last_became_active":"2024-03-21T08:41:08.014151+0000","last_became_peered":"2024-03-21T08:41:08.014151+0000","last_unstale":"2024-03-21T08:41:14.261730+0000","last_undegraded":"2024-03-21T08:41:14.261730+0000","last_fullsized":"2024-03-21T08:41:14.261730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260722+0000","last_change":"2024-03-21T08:41:08.019319+0000","last_active":"2024-03-21T08:41:14.260722+0000","last_peered":"2024-03-21T08:41:14.260722+0000","last_clean":"2024-03-21T08:41:14.260722+0000","last_became_active":"2024-03-21T08:41:08.019027+0000","last_became_peered":"2024-03-21T08:41:08.019027+0000","last_unstale":"2024-03-21T08:41:14.260722+0000","last_undegraded":"2024-03-21T08:41:14.260722+0000","last_fullsized":"2024-03-21T08:41:14.260722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161113+0000","last_change":"2024-03-21T08:41:08.018077+0000","last_active":"2024-03-21T08:41:13.161113+0000","last_peered":"2024-03-21T08:41:13.161113+0000","last_clean":"2024-03-21T08:41:13.161113+0000","last_became_active":"2024-03-21T08:41:08.017564+0000","last_became_peered":"2024-03-21T08:41:08.017564+0000","last_unstale":"2024-03-21T08:41:13.161113+0000","last_undegraded":"2024-03-21T08:41:13.161113+0000","last_fullsized":"2024-03-21T08:41:13.161113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026399+0000","last_change":"2024-03-21T08:41:08.018108+0000","last_active":"2024-03-21T08:41:13.026399+0000","last_peered":"2024-03-21T08:41:13.026399+0000","last_clean":"2024-03-21T08:41:13.026399+0000","last_became_active":"2024-03-21T08:41:08.017766+0000","last_became_peered":"2024-03-21T08:41:08.017766+0000","last_unstale":"2024-03-21T08:41:13.026399+0000","last_undegraded":"2024-03-21T08:41:13.026399+0000","last_fullsized":"2024-03-21T08:41:13.026399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737069+0000","last_change":"2024-03-21T08:41:08.021075+0000","last_active":"2024-03-21T08:41:13.737069+0000","last_peered":"2024-03-21T08:41:13.737069+0000","last_clean":"2024-03-21T08:41:13.737069+0000","last_became_active":"2024-03-21T08:41:08.020798+0000","last_became_peered":"2024-03-21T08:41:08.020798+0000","last_unstale":"2024-03-21T08:41:13.737069+0000","last_undegraded":"2024-03-21T08:41:13.737069+0000","last_fullsized":"2024-03-21T08:41:13.737069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:06.452855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025886+0000","last_change":"2024-03-21T08:41:08.008591+0000","last_active":"2024-03-21T08:41:13.025886+0000","last_peered":"2024-03-21T08:41:13.025886+0000","last_clean":"2024-03-21T08:41:13.025886+0000","last_became_active":"2024-03-21T08:41:08.008328+0000","last_became_peered":"2024-03-21T08:41:08.008328+0000","last_unstale":"2024-03-21T08:41:13.025886+0000","last_undegraded":"2024-03-21T08:41:13.025886+0000","last_fullsized":"2024-03-21T08:41:13.025886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:23:00.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261256+0000","last_change":"2024-03-21T08:41:08.023131+0000","last_active":"2024-03-21T08:41:14.261256+0000","last_peered":"2024-03-21T08:41:14.261256+0000","last_clean":"2024-03-21T08:41:14.261256+0000","last_became_active":"2024-03-21T08:41:08.022844+0000","last_became_peered":"2024-03-21T08:41:08.022844+0000","last_unstale":"2024-03-21T08:41:14.261256+0000","last_undegraded":"2024-03-21T08:41:14.261256+0000","last_fullsized":"2024-03-21T08:41:14.261256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:44.645908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260479+0000","last_change":"2024-03-21T08:41:08.008995+0000","last_active":"2024-03-21T08:41:14.260479+0000","last_peered":"2024-03-21T08:41:14.260479+0000","last_clean":"2024-03-21T08:41:14.260479+0000","last_became_active":"2024-03-21T08:41:08.008707+0000","last_became_peered":"2024-03-21T08:41:08.008707+0000","last_unstale":"2024-03-21T08:41:14.260479+0000","last_undegraded":"2024-03-21T08:41:14.260479+0000","last_fullsized":"2024-03-21T08:41:14.260479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261974+0000","last_change":"2024-03-21T08:41:08.007429+0000","last_active":"2024-03-21T08:41:14.261974+0000","last_peered":"2024-03-21T08:41:14.261974+0000","last_clean":"2024-03-21T08:41:14.261974+0000","last_became_active":"2024-03-21T08:41:08.007118+0000","last_became_peered":"2024-03-21T08:41:08.007118+0000","last_unstale":"2024-03-21T08:41:14.261974+0000","last_undegraded":"2024-03-21T08:41:14.261974+0000","last_fullsized":"2024-03-21T08:41:14.261974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160940+0000","last_change":"2024-03-21T08:41:08.002687+0000","last_active":"2024-03-21T08:41:13.160940+0000","last_peered":"2024-03-21T08:41:13.160940+0000","last_clean":"2024-03-21T08:41:13.160940+0000","last_became_active":"2024-03-21T08:41:08.002397+0000","last_became_peered":"2024-03-21T08:41:08.002397+0000","last_unstale":"2024-03-21T08:41:13.160940+0000","last_undegraded":"2024-03-21T08:41:13.160940+0000","last_fullsized":"2024-03-21T08:41:13.160940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:12:01.877634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262084+0000","last_change":"2024-03-21T08:41:08.011489+0000","last_active":"2024-03-21T08:41:14.262084+0000","last_peered":"2024-03-21T08:41:14.262084+0000","last_clean":"2024-03-21T08:41:14.262084+0000","last_became_active":"2024-03-21T08:41:08.011227+0000","last_became_peered":"2024-03-21T08:41:08.011227+0000","last_unstale":"2024-03-21T08:41:14.262084+0000","last_undegraded":"2024-03-21T08:41:14.262084+0000","last_fullsized":"2024-03-21T08:41:14.262084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027108+0000","last_change":"2024-03-21T08:41:08.010854+0000","last_active":"2024-03-21T08:41:13.027108+0000","last_peered":"2024-03-21T08:41:13.027108+0000","last_clean":"2024-03-21T08:41:13.027108+0000","last_became_active":"2024-03-21T08:41:08.010557+0000","last_became_peered":"2024-03-21T08:41:08.010557+0000","last_unstale":"2024-03-21T08:41:13.027108+0000","last_undegraded":"2024-03-21T08:41:13.027108+0000","last_fullsized":"2024-03-21T08:41:13.027108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:01.730277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026506+0000","last_change":"2024-03-21T08:41:08.014812+0000","last_active":"2024-03-21T08:41:13.026506+0000","last_peered":"2024-03-21T08:41:13.026506+0000","last_clean":"2024-03-21T08:41:13.026506+0000","last_became_active":"2024-03-21T08:41:08.014508+0000","last_became_peered":"2024-03-21T08:41:08.014508+0000","last_unstale":"2024-03-21T08:41:13.026506+0000","last_undegraded":"2024-03-21T08:41:13.026506+0000","last_fullsized":"2024-03-21T08:41:13.026506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026766+0000","last_change":"2024-03-21T08:41:08.013816+0000","last_active":"2024-03-21T08:41:13.026766+0000","last_peered":"2024-03-21T08:41:13.026766+0000","last_clean":"2024-03-21T08:41:13.026766+0000","last_became_active":"2024-03-21T08:41:08.013585+0000","last_became_peered":"2024-03-21T08:41:08.013585+0000","last_unstale":"2024-03-21T08:41:13.026766+0000","last_undegraded":"2024-03-21T08:41:13.026766+0000","last_fullsized":"2024-03-21T08:41:13.026766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737247+0000","last_change":"2024-03-21T08:41:08.031989+0000","last_active":"2024-03-21T08:41:13.737247+0000","last_peered":"2024-03-21T08:41:13.737247+0000","last_clean":"2024-03-21T08:41:13.737247+0000","last_became_active":"2024-03-21T08:41:08.031755+0000","last_became_peered":"2024-03-21T08:41:08.031755+0000","last_unstale":"2024-03-21T08:41:13.737247+0000","last_undegraded":"2024-03-21T08:41:13.737247+0000","last_fullsized":"2024-03-21T08:41:13.737247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:37.172205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261442+0000","last_change":"2024-03-21T08:41:08.015070+0000","last_active":"2024-03-21T08:41:14.261442+0000","last_peered":"2024-03-21T08:41:14.261442+0000","last_clean":"2024-03-21T08:41:14.261442+0000","last_became_active":"2024-03-21T08:41:08.012083+0000","last_became_peered":"2024-03-21T08:41:08.012083+0000","last_unstale":"2024-03-21T08:41:14.261442+0000","last_undegraded":"2024-03-21T08:41:14.261442+0000","last_fullsized":"2024-03-21T08:41:14.261442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736535+0000","last_change":"2024-03-21T08:41:08.020740+0000","last_active":"2024-03-21T08:41:13.736535+0000","last_peered":"2024-03-21T08:41:13.736535+0000","last_clean":"2024-03-21T08:41:13.736535+0000","last_became_active":"2024-03-21T08:41:08.020396+0000","last_became_peered":"2024-03-21T08:41:08.020396+0000","last_unstale":"2024-03-21T08:41:13.736535+0000","last_undegraded":"2024-03-21T08:41:13.736535+0000","last_fullsized":"2024-03-21T08:41:13.736535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:59:37.815273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260654+0000","last_change":"2024-03-21T08:41:08.010314+0000","last_active":"2024-03-21T08:41:14.260654+0000","last_peered":"2024-03-21T08:41:14.260654+0000","last_clean":"2024-03-21T08:41:14.260654+0000","last_became_active":"2024-03-21T08:41:08.009693+0000","last_became_peered":"2024-03-21T08:41:08.009693+0000","last_unstale":"2024-03-21T08:41:14.260654+0000","last_undegraded":"2024-03-21T08:41:14.260654+0000","last_fullsized":"2024-03-21T08:41:14.260654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736584+0000","last_change":"2024-03-21T08:41:08.020466+0000","last_active":"2024-03-21T08:41:13.736584+0000","last_peered":"2024-03-21T08:41:13.736584+0000","last_clean":"2024-03-21T08:41:13.736584+0000","last_became_active":"2024-03-21T08:41:08.020169+0000","last_became_peered":"2024-03-21T08:41:08.020169+0000","last_unstale":"2024-03-21T08:41:13.736584+0000","last_undegraded":"2024-03-21T08:41:13.736584+0000","last_fullsized":"2024-03-21T08:41:13.736584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:00.508480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260900+0000","last_change":"2024-03-21T08:41:08.014151+0000","last_active":"2024-03-21T08:41:14.260900+0000","last_peered":"2024-03-21T08:41:14.260900+0000","last_clean":"2024-03-21T08:41:14.260900+0000","last_became_active":"2024-03-21T08:41:08.013319+0000","last_became_peered":"2024-03-21T08:41:08.013319+0000","last_unstale":"2024-03-21T08:41:14.260900+0000","last_undegraded":"2024-03-21T08:41:14.260900+0000","last_fullsized":"2024-03-21T08:41:14.260900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:16.725366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161103+0000","last_change":"2024-03-21T08:41:08.002931+0000","last_active":"2024-03-21T08:41:13.161103+0000","last_peered":"2024-03-21T08:41:13.161103+0000","last_clean":"2024-03-21T08:41:13.161103+0000","last_became_active":"2024-03-21T08:41:08.001953+0000","last_became_peered":"2024-03-21T08:41:08.001953+0000","last_unstale":"2024-03-21T08:41:13.161103+0000","last_undegraded":"2024-03-21T08:41:13.161103+0000","last_fullsized":"2024-03-21T08:41:13.161103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263385+0000","last_change":"2024-03-21T08:41:08.005883+0000","last_active":"2024-03-21T08:41:14.263385+0000","last_peered":"2024-03-21T08:41:14.263385+0000","last_clean":"2024-03-21T08:41:14.263385+0000","last_became_active":"2024-03-21T08:41:08.005503+0000","last_became_peered":"2024-03-21T08:41:08.005503+0000","last_unstale":"2024-03-21T08:41:14.263385+0000","last_undegraded":"2024-03-21T08:41:14.263385+0000","last_fullsized":"2024-03-21T08:41:14.263385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737138+0000","last_change":"2024-03-21T08:41:08.030286+0000","last_active":"2024-03-21T08:41:13.737138+0000","last_peered":"2024-03-21T08:41:13.737138+0000","last_clean":"2024-03-21T08:41:13.737138+0000","last_became_active":"2024-03-21T08:41:08.029830+0000","last_became_peered":"2024-03-21T08:41:08.029830+0000","last_unstale":"2024-03-21T08:41:13.737138+0000","last_undegraded":"2024-03-21T08:41:13.737138+0000","last_fullsized":"2024-03-21T08:41:13.737138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:31.510333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509451,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":693167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":102377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":693223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:41:31.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:41:32.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.073+0000 7f323c400640 1 Processor -- start 2024-03-21T08:41:32.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.073+0000 7f323c400640 1 -- start start 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32340ce490 0x7f323415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f323415ba70 0x7f32340ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3234071980 0x7f3234071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340cedb0 con 0x7f3234071980 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323415bed0 con 0x7f323415ba70 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323415ebf0 con 0x7f32340ce490 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3234071980 0x7f3234071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3234071980 0x7f3234071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55628/0 (socket says 172.21.15.28:55628) 2024-03-21T08:41:32.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 -- 172.21.15.28:0/5173926 learned_addr learned my addr 172.21.15.28:0/5173926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f3239000640 1 --2- 172.21.15.28:0/5173926 >> v2:172.21.15.151:3300/0 conn(0x7f323415ba70 0x7f32340ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f3239a00640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32340ce490 0x7f323415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 -- 172.21.15.28:0/5173926 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32340ce490 msgr2=0x7f323415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32340ce490 0x7f323415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 -- 172.21.15.28:0/5173926 >> v2:172.21.15.151:3300/0 conn(0x7f323415ba70 msgr2=0x7f32340ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 --2- 172.21.15.28:0/5173926 >> v2:172.21.15.151:3300/0 conn(0x7f323415ba70 0x7f32340ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.077+0000 7f323a400640 1 -- 172.21.15.28:0/5173926 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323415eed0 con 0x7f3234071980 2024-03-21T08:41:32.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323a400640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234071dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f32340b25a0 tx=0x7f322c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a215654fcfea6b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f3233e00640 1 -- 172.21.15.28:0/5173926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c0a55d0 con 0x7f3234071980 2024-03-21T08:41:32.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f3233e00640 1 -- 172.21.15.28:0/5173926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c0a5770 con 0x7f3234071980 2024-03-21T08:41:32.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f3233e00640 1 -- 172.21.15.28:0/5173926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c0a5b40 con 0x7f3234071980 2024-03-21T08:41:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- 172.21.15.28:0/5173926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 msgr2=0x7f3234071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234071dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f32340b25a0 tx=0x7f322c01c960 comp rx=0 tx=0).stop 2024-03-21T08:41:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- 172.21.15.28:0/5173926 shutdown_connections 2024-03-21T08:41:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234071dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 --2- 172.21.15.28:0/5173926 >> v2:172.21.15.151:3300/0 conn(0x7f323415ba70 0x7f32340ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 --2- 172.21.15.28:0/5173926 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32340ce490 0x7f323415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- 172.21.15.28:0/5173926 >> 172.21.15.28:0/5173926 conn(0x7f32340cc230 msgr2=0x7f32340cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- 172.21.15.28:0/5173926 shutdown_connections 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- 172.21.15.28:0/5173926 wait complete. 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 Processor -- start 2024-03-21T08:41:32.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 -- start start 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.081+0000 7f323c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 0x7f323414b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 0x7f3234153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3234155390 con 0x7f3234071980 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3234072720 con 0x7f32340ce490 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32340cdb90 con 0x7f323415ba70 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55642/0 (socket says 172.21.15.28:55642) 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 -- 172.21.15.28:0/2671934909 learned_addr learned my addr 172.21.15.28:0/2671934909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239000640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 0x7f323414b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323a400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 0x7f3234153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 -- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 msgr2=0x7f3234153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 0x7f3234153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 -- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 msgr2=0x7f323414b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 0x7f323414b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 -- 172.21.15.28:0/2671934909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323414bf70 con 0x7f3234071980 2024-03-21T08:41:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239000640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 0x7f323414b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323a400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 0x7f3234153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3239a00640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f322800e8b0 tx=0x7f322800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228017070 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322800c940 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800cd10 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323414c250 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323414c810 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323404f410 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f322801a070 con 0x7f3234071980 2024-03-21T08:41:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.085+0000 7f3231600640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 0x7f321406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.089+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3228086670 con 0x7f3234071980 2024-03-21T08:41:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.089+0000 7f3239000640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 0x7f321406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.089+0000 7f3239000640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 0x7f321406a010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3234153a50 tx=0x7f3220008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.093+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3228056ca0 con 0x7f3234071980 2024-03-21T08:41:32.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.253+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f323414cd60 con 0x7f3214067b50 2024-03-21T08:41:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.261+0000 7f3231600640 1 -- 172.21.15.28:0/2671934909 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332538 (secure 0 0 0) 0x7f323414cd60 con 0x7f3214067b50 2024-03-21T08:41:32.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 msgr2=0x7f321406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 0x7f321406a010 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3234153a50 tx=0x7f3220008040 comp rx=0 tx=0).stop 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 msgr2=0x7f3234137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f322800e8b0 tx=0x7f322800ed70 comp rx=0 tx=0).stop 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 shutdown_connections 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3214067b50 0x7f321406a010 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f323415ba70 0x7f3234153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32340ce490 0x7f323414b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 --2- 172.21.15.28:0/2671934909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3234071980 0x7f3234137bf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 >> 172.21.15.28:0/2671934909 conn(0x7f32340cc230 msgr2=0x7f323406d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 shutdown_connections 2024-03-21T08:41:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.265+0000 7f323c400640 1 -- 172.21.15.28:0/2671934909 wait complete. 2024-03-21T08:41:32.288 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-21T08:41:31.724067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231892,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742828,"statfs":{"total":773094113280,"available":772856655872,"internally_reserved":0,"allocated":3317760,"data_stored":2007752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161451+0000","last_change":"2024-03-21T08:41:07.021438+0000","last_active":"2024-03-21T08:41:13.161451+0000","last_peered":"2024-03-21T08:41:13.161451+0000","last_clean":"2024-03-21T08:41:13.161451+0000","last_became_active":"2024-03-21T08:41:07.021142+0000","last_became_peered":"2024-03-21T08:41:07.021142+0000","last_unstale":"2024-03-21T08:41:13.161451+0000","last_undegraded":"2024-03-21T08:41:13.161451+0000","last_fullsized":"2024-03-21T08:41:13.161451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:15:32.936778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262630+0000","last_change":"2024-03-21T08:41:08.010424+0000","last_active":"2024-03-21T08:41:14.262630+0000","last_peered":"2024-03-21T08:41:14.262630+0000","last_clean":"2024-03-21T08:41:14.262630+0000","last_became_active":"2024-03-21T08:41:08.010169+0000","last_became_peered":"2024-03-21T08:41:08.010169+0000","last_unstale":"2024-03-21T08:41:14.262630+0000","last_undegraded":"2024-03-21T08:41:14.262630+0000","last_fullsized":"2024-03-21T08:41:14.262630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261510+0000","last_change":"2024-03-21T08:41:12.340891+0000","last_active":"2024-03-21T08:41:14.261510+0000","last_peered":"2024-03-21T08:41:14.261510+0000","last_clean":"2024-03-21T08:41:14.261510+0000","last_became_active":"2024-03-21T08:41:12.340181+0000","last_became_peered":"2024-03-21T08:41:12.340181+0000","last_unstale":"2024-03-21T08:41:14.261510+0000","last_undegraded":"2024-03-21T08:41:14.261510+0000","last_fullsized":"2024-03-21T08:41:14.261510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262757+0000","last_change":"2024-03-21T08:41:07.009640+0000","last_active":"2024-03-21T08:41:14.262757+0000","last_peered":"2024-03-21T08:41:14.262757+0000","last_clean":"2024-03-21T08:41:14.262757+0000","last_became_active":"2024-03-21T08:41:07.009390+0000","last_became_peered":"2024-03-21T08:41:07.009390+0000","last_unstale":"2024-03-21T08:41:14.262757+0000","last_undegraded":"2024-03-21T08:41:14.262757+0000","last_fullsized":"2024-03-21T08:41:14.262757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161250+0000","last_change":"2024-03-21T08:41:08.010008+0000","last_active":"2024-03-21T08:41:13.161250+0000","last_peered":"2024-03-21T08:41:13.161250+0000","last_clean":"2024-03-21T08:41:13.161250+0000","last_became_active":"2024-03-21T08:41:08.008163+0000","last_became_peered":"2024-03-21T08:41:08.008163+0000","last_unstale":"2024-03-21T08:41:13.161250+0000","last_undegraded":"2024-03-21T08:41:13.161250+0000","last_fullsized":"2024-03-21T08:41:13.161250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025449+0000","last_change":"2024-03-21T08:41:12.049570+0000","last_active":"2024-03-21T08:41:13.025449+0000","last_peered":"2024-03-21T08:41:13.025449+0000","last_clean":"2024-03-21T08:41:13.025449+0000","last_became_active":"2024-03-21T08:41:12.048903+0000","last_became_peered":"2024-03-21T08:41:12.048903+0000","last_unstale":"2024-03-21T08:41:13.025449+0000","last_undegraded":"2024-03-21T08:41:13.025449+0000","last_fullsized":"2024-03-21T08:41:13.025449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:10.935038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736993+0000","last_change":"2024-03-21T08:41:07.022988+0000","last_active":"2024-03-21T08:41:13.736993+0000","last_peered":"2024-03-21T08:41:13.736993+0000","last_clean":"2024-03-21T08:41:13.736993+0000","last_became_active":"2024-03-21T08:41:07.022643+0000","last_became_peered":"2024-03-21T08:41:07.022643+0000","last_unstale":"2024-03-21T08:41:13.736993+0000","last_undegraded":"2024-03-21T08:41:13.736993+0000","last_fullsized":"2024-03-21T08:41:13.736993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260935+0000","last_change":"2024-03-21T08:41:08.008250+0000","last_active":"2024-03-21T08:41:14.260935+0000","last_peered":"2024-03-21T08:41:14.260935+0000","last_clean":"2024-03-21T08:41:14.260935+0000","last_became_active":"2024-03-21T08:41:08.007638+0000","last_became_peered":"2024-03-21T08:41:08.007638+0000","last_unstale":"2024-03-21T08:41:14.260935+0000","last_undegraded":"2024-03-21T08:41:14.260935+0000","last_fullsized":"2024-03-21T08:41:14.260935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:14.189872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027230+0000","last_change":"2024-03-21T08:41:12.051136+0000","last_active":"2024-03-21T08:41:13.027230+0000","last_peered":"2024-03-21T08:41:13.027230+0000","last_clean":"2024-03-21T08:41:13.027230+0000","last_became_active":"2024-03-21T08:41:12.050706+0000","last_became_peered":"2024-03-21T08:41:12.050706+0000","last_unstale":"2024-03-21T08:41:13.027230+0000","last_undegraded":"2024-03-21T08:41:13.027230+0000","last_fullsized":"2024-03-21T08:41:13.027230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:28:03.320473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736865+0000","last_change":"2024-03-21T08:41:07.021316+0000","last_active":"2024-03-21T08:41:13.736865+0000","last_peered":"2024-03-21T08:41:13.736865+0000","last_clean":"2024-03-21T08:41:13.736865+0000","last_became_active":"2024-03-21T08:41:07.021039+0000","last_became_peered":"2024-03-21T08:41:07.021039+0000","last_unstale":"2024-03-21T08:41:13.736865+0000","last_undegraded":"2024-03-21T08:41:13.736865+0000","last_fullsized":"2024-03-21T08:41:13.736865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261060+0000","last_change":"2024-03-21T08:41:08.004273+0000","last_active":"2024-03-21T08:41:14.261060+0000","last_peered":"2024-03-21T08:41:14.261060+0000","last_clean":"2024-03-21T08:41:14.261060+0000","last_became_active":"2024-03-21T08:41:08.003800+0000","last_became_peered":"2024-03-21T08:41:08.003800+0000","last_unstale":"2024-03-21T08:41:14.261060+0000","last_undegraded":"2024-03-21T08:41:14.261060+0000","last_fullsized":"2024-03-21T08:41:14.261060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027771+0000","last_change":"2024-03-21T08:41:12.046634+0000","last_active":"2024-03-21T08:41:13.027771+0000","last_peered":"2024-03-21T08:41:13.027771+0000","last_clean":"2024-03-21T08:41:13.027771+0000","last_became_active":"2024-03-21T08:41:12.046224+0000","last_became_peered":"2024-03-21T08:41:12.046224+0000","last_unstale":"2024-03-21T08:41:13.027771+0000","last_undegraded":"2024-03-21T08:41:13.027771+0000","last_fullsized":"2024-03-21T08:41:13.027771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:20.757184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025520+0000","last_change":"2024-03-21T08:41:08.009427+0000","last_active":"2024-03-21T08:41:13.025520+0000","last_peered":"2024-03-21T08:41:13.025520+0000","last_clean":"2024-03-21T08:41:13.025520+0000","last_became_active":"2024-03-21T08:41:08.009082+0000","last_became_peered":"2024-03-21T08:41:08.009082+0000","last_unstale":"2024-03-21T08:41:13.025520+0000","last_undegraded":"2024-03-21T08:41:13.025520+0000","last_fullsized":"2024-03-21T08:41:13.025520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:54:07.174703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736774+0000","last_change":"2024-03-21T08:41:07.019968+0000","last_active":"2024-03-21T08:41:13.736774+0000","last_peered":"2024-03-21T08:41:13.736774+0000","last_clean":"2024-03-21T08:41:13.736774+0000","last_became_active":"2024-03-21T08:41:07.019039+0000","last_became_peered":"2024-03-21T08:41:07.019039+0000","last_unstale":"2024-03-21T08:41:13.736774+0000","last_undegraded":"2024-03-21T08:41:13.736774+0000","last_fullsized":"2024-03-21T08:41:13.736774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027696+0000","last_change":"2024-03-21T08:41:12.053357+0000","last_active":"2024-03-21T08:41:13.027696+0000","last_peered":"2024-03-21T08:41:13.027696+0000","last_clean":"2024-03-21T08:41:13.027696+0000","last_became_active":"2024-03-21T08:41:12.053045+0000","last_became_peered":"2024-03-21T08:41:12.053045+0000","last_unstale":"2024-03-21T08:41:13.027696+0000","last_undegraded":"2024-03-21T08:41:13.027696+0000","last_fullsized":"2024-03-21T08:41:13.027696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:52.060002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262174+0000","last_change":"2024-03-21T08:41:08.014032+0000","last_active":"2024-03-21T08:41:14.262174+0000","last_peered":"2024-03-21T08:41:14.262174+0000","last_clean":"2024-03-21T08:41:14.262174+0000","last_became_active":"2024-03-21T08:41:08.013716+0000","last_became_peered":"2024-03-21T08:41:08.013716+0000","last_unstale":"2024-03-21T08:41:14.262174+0000","last_undegraded":"2024-03-21T08:41:14.262174+0000","last_fullsized":"2024-03-21T08:41:14.262174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262296+0000","last_change":"2024-03-21T08:41:07.008347+0000","last_active":"2024-03-21T08:41:14.262296+0000","last_peered":"2024-03-21T08:41:14.262296+0000","last_clean":"2024-03-21T08:41:14.262296+0000","last_became_active":"2024-03-21T08:41:07.005441+0000","last_became_peered":"2024-03-21T08:41:07.005441+0000","last_unstale":"2024-03-21T08:41:14.262296+0000","last_undegraded":"2024-03-21T08:41:14.262296+0000","last_fullsized":"2024-03-21T08:41:14.262296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:22:59.344972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025374+0000","last_change":"2024-03-21T08:41:12.051147+0000","last_active":"2024-03-21T08:41:13.025374+0000","last_peered":"2024-03-21T08:41:13.025374+0000","last_clean":"2024-03-21T08:41:13.025374+0000","last_became_active":"2024-03-21T08:41:12.050744+0000","last_became_peered":"2024-03-21T08:41:12.050744+0000","last_unstale":"2024-03-21T08:41:13.025374+0000","last_undegraded":"2024-03-21T08:41:13.025374+0000","last_fullsized":"2024-03-21T08:41:13.025374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:15.657762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261625+0000","last_change":"2024-03-21T08:41:08.003261+0000","last_active":"2024-03-21T08:41:14.261625+0000","last_peered":"2024-03-21T08:41:14.261625+0000","last_clean":"2024-03-21T08:41:14.261625+0000","last_became_active":"2024-03-21T08:41:08.002934+0000","last_became_peered":"2024-03-21T08:41:08.002934+0000","last_unstale":"2024-03-21T08:41:14.261625+0000","last_undegraded":"2024-03-21T08:41:14.261625+0000","last_fullsized":"2024-03-21T08:41:14.261625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262016+0000","last_change":"2024-03-21T08:41:07.020788+0000","last_active":"2024-03-21T08:41:14.262016+0000","last_peered":"2024-03-21T08:41:14.262016+0000","last_clean":"2024-03-21T08:41:14.262016+0000","last_became_active":"2024-03-21T08:41:07.020341+0000","last_became_peered":"2024-03-21T08:41:07.020341+0000","last_unstale":"2024-03-21T08:41:14.262016+0000","last_undegraded":"2024-03-21T08:41:14.262016+0000","last_fullsized":"2024-03-21T08:41:14.262016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:45.135431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263514+0000","last_change":"2024-03-21T08:41:12.340482+0000","last_active":"2024-03-21T08:41:14.263514+0000","last_peered":"2024-03-21T08:41:14.263514+0000","last_clean":"2024-03-21T08:41:14.263514+0000","last_became_active":"2024-03-21T08:41:12.340076+0000","last_became_peered":"2024-03-21T08:41:12.340076+0000","last_unstale":"2024-03-21T08:41:14.263514+0000","last_undegraded":"2024-03-21T08:41:14.263514+0000","last_fullsized":"2024-03-21T08:41:14.263514+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:10.484672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027247+0000","last_change":"2024-03-21T08:41:08.002403+0000","last_active":"2024-03-21T08:41:13.027247+0000","last_peered":"2024-03-21T08:41:13.027247+0000","last_clean":"2024-03-21T08:41:13.027247+0000","last_became_active":"2024-03-21T08:41:08.001933+0000","last_became_peered":"2024-03-21T08:41:08.001933+0000","last_unstale":"2024-03-21T08:41:13.027247+0000","last_undegraded":"2024-03-21T08:41:13.027247+0000","last_fullsized":"2024-03-21T08:41:13.027247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:40.121158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736405+0000","last_change":"2024-03-21T08:41:07.019255+0000","last_active":"2024-03-21T08:41:13.736405+0000","last_peered":"2024-03-21T08:41:13.736405+0000","last_clean":"2024-03-21T08:41:13.736405+0000","last_became_active":"2024-03-21T08:41:07.018958+0000","last_became_peered":"2024-03-21T08:41:07.018958+0000","last_unstale":"2024-03-21T08:41:13.736405+0000","last_undegraded":"2024-03-21T08:41:13.736405+0000","last_fullsized":"2024-03-21T08:41:13.736405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027872+0000","last_change":"2024-03-21T08:41:12.049104+0000","last_active":"2024-03-21T08:41:13.027872+0000","last_peered":"2024-03-21T08:41:13.027872+0000","last_clean":"2024-03-21T08:41:13.027872+0000","last_became_active":"2024-03-21T08:41:12.048724+0000","last_became_peered":"2024-03-21T08:41:12.048724+0000","last_unstale":"2024-03-21T08:41:13.027872+0000","last_undegraded":"2024-03-21T08:41:13.027872+0000","last_fullsized":"2024-03-21T08:41:13.027872+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161500+0000","last_change":"2024-03-21T08:41:08.006976+0000","last_active":"2024-03-21T08:41:13.161500+0000","last_peered":"2024-03-21T08:41:13.161500+0000","last_clean":"2024-03-21T08:41:13.161500+0000","last_became_active":"2024-03-21T08:41:08.006565+0000","last_became_peered":"2024-03-21T08:41:08.006565+0000","last_unstale":"2024-03-21T08:41:13.161500+0000","last_undegraded":"2024-03-21T08:41:13.161500+0000","last_fullsized":"2024-03-21T08:41:13.161500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026772+0000","last_change":"2024-03-21T08:41:07.014131+0000","last_active":"2024-03-21T08:41:13.026772+0000","last_peered":"2024-03-21T08:41:13.026772+0000","last_clean":"2024-03-21T08:41:13.026772+0000","last_became_active":"2024-03-21T08:41:07.013789+0000","last_became_peered":"2024-03-21T08:41:07.013789+0000","last_unstale":"2024-03-21T08:41:13.026772+0000","last_undegraded":"2024-03-21T08:41:13.026772+0000","last_fullsized":"2024-03-21T08:41:13.026772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027502+0000","last_change":"2024-03-21T08:41:12.336087+0000","last_active":"2024-03-21T08:41:13.027502+0000","last_peered":"2024-03-21T08:41:13.027502+0000","last_clean":"2024-03-21T08:41:13.027502+0000","last_became_active":"2024-03-21T08:41:12.335722+0000","last_became_peered":"2024-03-21T08:41:12.335722+0000","last_unstale":"2024-03-21T08:41:13.027502+0000","last_undegraded":"2024-03-21T08:41:13.027502+0000","last_fullsized":"2024-03-21T08:41:13.027502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:47.106105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262994+0000","last_change":"2024-03-21T08:41:08.021861+0000","last_active":"2024-03-21T08:41:14.262994+0000","last_peered":"2024-03-21T08:41:14.262994+0000","last_clean":"2024-03-21T08:41:14.262994+0000","last_became_active":"2024-03-21T08:41:08.021564+0000","last_became_peered":"2024-03-21T08:41:08.021564+0000","last_unstale":"2024-03-21T08:41:14.262994+0000","last_undegraded":"2024-03-21T08:41:14.262994+0000","last_fullsized":"2024-03-21T08:41:14.262994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:56:43.074287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263418+0000","last_change":"2024-03-21T08:41:07.010549+0000","last_active":"2024-03-21T08:41:14.263418+0000","last_peered":"2024-03-21T08:41:14.263418+0000","last_clean":"2024-03-21T08:41:14.263418+0000","last_became_active":"2024-03-21T08:41:07.010129+0000","last_became_peered":"2024-03-21T08:41:07.010129+0000","last_unstale":"2024-03-21T08:41:14.263418+0000","last_undegraded":"2024-03-21T08:41:14.263418+0000","last_fullsized":"2024-03-21T08:41:14.263418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261368+0000","last_change":"2024-03-21T08:41:12.340111+0000","last_active":"2024-03-21T08:41:14.261368+0000","last_peered":"2024-03-21T08:41:14.261368+0000","last_clean":"2024-03-21T08:41:14.261368+0000","last_became_active":"2024-03-21T08:41:12.339714+0000","last_became_peered":"2024-03-21T08:41:12.339714+0000","last_unstale":"2024-03-21T08:41:14.261368+0000","last_undegraded":"2024-03-21T08:41:14.261368+0000","last_fullsized":"2024-03-21T08:41:14.261368+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:30:09.350945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262982+0000","last_change":"2024-03-21T08:41:08.005617+0000","last_active":"2024-03-21T08:41:14.262982+0000","last_peered":"2024-03-21T08:41:14.262982+0000","last_clean":"2024-03-21T08:41:14.262982+0000","last_became_active":"2024-03-21T08:41:08.005254+0000","last_became_peered":"2024-03-21T08:41:08.005254+0000","last_unstale":"2024-03-21T08:41:14.262982+0000","last_undegraded":"2024-03-21T08:41:14.262982+0000","last_fullsized":"2024-03-21T08:41:14.262982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737093+0000","last_change":"2024-03-21T08:41:07.019857+0000","last_active":"2024-03-21T08:41:13.737093+0000","last_peered":"2024-03-21T08:41:13.737093+0000","last_clean":"2024-03-21T08:41:13.737093+0000","last_became_active":"2024-03-21T08:41:07.019553+0000","last_became_peered":"2024-03-21T08:41:07.019553+0000","last_unstale":"2024-03-21T08:41:13.737093+0000","last_undegraded":"2024-03-21T08:41:13.737093+0000","last_fullsized":"2024-03-21T08:41:13.737093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:35:34.563261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161134+0000","last_change":"2024-03-21T08:41:12.335655+0000","last_active":"2024-03-21T08:41:13.161134+0000","last_peered":"2024-03-21T08:41:13.161134+0000","last_clean":"2024-03-21T08:41:13.161134+0000","last_became_active":"2024-03-21T08:41:12.335219+0000","last_became_peered":"2024-03-21T08:41:12.335219+0000","last_unstale":"2024-03-21T08:41:13.161134+0000","last_undegraded":"2024-03-21T08:41:13.161134+0000","last_fullsized":"2024-03-21T08:41:13.161134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:47:03.094208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261158+0000","last_change":"2024-03-21T08:41:08.016121+0000","last_active":"2024-03-21T08:41:14.261158+0000","last_peered":"2024-03-21T08:41:14.261158+0000","last_clean":"2024-03-21T08:41:14.261158+0000","last_became_active":"2024-03-21T08:41:08.015237+0000","last_became_peered":"2024-03-21T08:41:08.015237+0000","last_unstale":"2024-03-21T08:41:14.261158+0000","last_undegraded":"2024-03-21T08:41:14.261158+0000","last_fullsized":"2024-03-21T08:41:14.261158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:48.028184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263069+0000","last_change":"2024-03-21T08:41:07.013927+0000","last_active":"2024-03-21T08:41:14.263069+0000","last_peered":"2024-03-21T08:41:14.263069+0000","last_clean":"2024-03-21T08:41:14.263069+0000","last_became_active":"2024-03-21T08:41:07.013595+0000","last_became_peered":"2024-03-21T08:41:07.013595+0000","last_unstale":"2024-03-21T08:41:14.263069+0000","last_undegraded":"2024-03-21T08:41:14.263069+0000","last_fullsized":"2024-03-21T08:41:14.263069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262181+0000","last_change":"2024-03-21T08:41:12.056397+0000","last_active":"2024-03-21T08:41:14.262181+0000","last_peered":"2024-03-21T08:41:14.262181+0000","last_clean":"2024-03-21T08:41:14.262181+0000","last_became_active":"2024-03-21T08:41:12.054349+0000","last_became_peered":"2024-03-21T08:41:12.054349+0000","last_unstale":"2024-03-21T08:41:14.262181+0000","last_undegraded":"2024-03-21T08:41:14.262181+0000","last_fullsized":"2024-03-21T08:41:14.262181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027232+0000","last_change":"2024-03-21T08:41:07.997069+0000","last_active":"2024-03-21T08:41:13.027232+0000","last_peered":"2024-03-21T08:41:13.027232+0000","last_clean":"2024-03-21T08:41:13.027232+0000","last_became_active":"2024-03-21T08:41:07.996691+0000","last_became_peered":"2024-03-21T08:41:07.996691+0000","last_unstale":"2024-03-21T08:41:13.027232+0000","last_undegraded":"2024-03-21T08:41:13.027232+0000","last_fullsized":"2024-03-21T08:41:13.027232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:59.860172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260862+0000","last_change":"2024-03-21T08:41:07.004740+0000","last_active":"2024-03-21T08:41:14.260862+0000","last_peered":"2024-03-21T08:41:14.260862+0000","last_clean":"2024-03-21T08:41:14.260862+0000","last_became_active":"2024-03-21T08:41:07.004315+0000","last_became_peered":"2024-03-21T08:41:07.004315+0000","last_unstale":"2024-03-21T08:41:14.260862+0000","last_undegraded":"2024-03-21T08:41:14.260862+0000","last_fullsized":"2024-03-21T08:41:14.260862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:10:58.854457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263063+0000","last_change":"2024-03-21T08:41:12.049583+0000","last_active":"2024-03-21T08:41:14.263063+0000","last_peered":"2024-03-21T08:41:14.263063+0000","last_clean":"2024-03-21T08:41:14.263063+0000","last_became_active":"2024-03-21T08:41:12.048879+0000","last_became_peered":"2024-03-21T08:41:12.048879+0000","last_unstale":"2024-03-21T08:41:14.263063+0000","last_undegraded":"2024-03-21T08:41:14.263063+0000","last_fullsized":"2024-03-21T08:41:14.263063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:41.204023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261754+0000","last_change":"2024-03-21T08:41:08.006302+0000","last_active":"2024-03-21T08:41:14.261754+0000","last_peered":"2024-03-21T08:41:14.261754+0000","last_clean":"2024-03-21T08:41:14.261754+0000","last_became_active":"2024-03-21T08:41:08.005255+0000","last_became_peered":"2024-03-21T08:41:08.005255+0000","last_unstale":"2024-03-21T08:41:14.261754+0000","last_undegraded":"2024-03-21T08:41:14.261754+0000","last_fullsized":"2024-03-21T08:41:14.261754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025568+0000","last_change":"2024-03-21T08:41:07.004853+0000","last_active":"2024-03-21T08:41:13.025568+0000","last_peered":"2024-03-21T08:41:13.025568+0000","last_clean":"2024-03-21T08:41:13.025568+0000","last_became_active":"2024-03-21T08:41:07.004447+0000","last_became_peered":"2024-03-21T08:41:07.004447+0000","last_unstale":"2024-03-21T08:41:13.025568+0000","last_undegraded":"2024-03-21T08:41:13.025568+0000","last_fullsized":"2024-03-21T08:41:13.025568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263561+0000","last_change":"2024-03-21T08:41:12.057098+0000","last_active":"2024-03-21T08:41:14.263561+0000","last_peered":"2024-03-21T08:41:14.263561+0000","last_clean":"2024-03-21T08:41:14.263561+0000","last_became_active":"2024-03-21T08:41:12.056306+0000","last_became_peered":"2024-03-21T08:41:12.056306+0000","last_unstale":"2024-03-21T08:41:14.263561+0000","last_undegraded":"2024-03-21T08:41:14.263561+0000","last_fullsized":"2024-03-21T08:41:14.263561+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:00:00.666748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160776+0000","last_change":"2024-03-21T08:41:07.995187+0000","last_active":"2024-03-21T08:41:13.160776+0000","last_peered":"2024-03-21T08:41:13.160776+0000","last_clean":"2024-03-21T08:41:13.160776+0000","last_became_active":"2024-03-21T08:41:07.994882+0000","last_became_peered":"2024-03-21T08:41:07.994882+0000","last_unstale":"2024-03-21T08:41:13.160776+0000","last_undegraded":"2024-03-21T08:41:13.160776+0000","last_fullsized":"2024-03-21T08:41:13.160776+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:55:27.177399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736421+0000","last_change":"2024-03-21T08:41:07.019906+0000","last_active":"2024-03-21T08:41:13.736421+0000","last_peered":"2024-03-21T08:41:13.736421+0000","last_clean":"2024-03-21T08:41:13.736421+0000","last_became_active":"2024-03-21T08:41:07.019585+0000","last_became_peered":"2024-03-21T08:41:07.019585+0000","last_unstale":"2024-03-21T08:41:13.736421+0000","last_undegraded":"2024-03-21T08:41:13.736421+0000","last_fullsized":"2024-03-21T08:41:13.736421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161357+0000","last_change":"2024-03-21T08:41:12.337710+0000","last_active":"2024-03-21T08:41:13.161357+0000","last_peered":"2024-03-21T08:41:13.161357+0000","last_clean":"2024-03-21T08:41:13.161357+0000","last_became_active":"2024-03-21T08:41:12.336270+0000","last_became_peered":"2024-03-21T08:41:12.336270+0000","last_unstale":"2024-03-21T08:41:13.161357+0000","last_undegraded":"2024-03-21T08:41:13.161357+0000","last_fullsized":"2024-03-21T08:41:13.161357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:11.100710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261759+0000","last_change":"2024-03-21T08:41:08.015395+0000","last_active":"2024-03-21T08:41:14.261759+0000","last_peered":"2024-03-21T08:41:14.261759+0000","last_clean":"2024-03-21T08:41:14.261759+0000","last_became_active":"2024-03-21T08:41:08.015091+0000","last_became_peered":"2024-03-21T08:41:08.015091+0000","last_unstale":"2024-03-21T08:41:14.261759+0000","last_undegraded":"2024-03-21T08:41:14.261759+0000","last_fullsized":"2024-03-21T08:41:14.261759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260786+0000","last_change":"2024-03-21T08:41:07.008427+0000","last_active":"2024-03-21T08:41:14.260786+0000","last_peered":"2024-03-21T08:41:14.260786+0000","last_clean":"2024-03-21T08:41:14.260786+0000","last_became_active":"2024-03-21T08:41:07.008115+0000","last_became_peered":"2024-03-21T08:41:07.008115+0000","last_unstale":"2024-03-21T08:41:14.260786+0000","last_undegraded":"2024-03-21T08:41:14.260786+0000","last_fullsized":"2024-03-21T08:41:14.260786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:25:26.626266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736516+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:41:13.736516+0000","last_peered":"2024-03-21T08:41:13.736516+0000","last_clean":"2024-03-21T08:41:13.736516+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:41:13.736516+0000","last_undegraded":"2024-03-21T08:41:13.736516+0000","last_fullsized":"2024-03-21T08:41:13.736516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026534+0000","last_change":"2024-03-21T08:41:12.334882+0000","last_active":"2024-03-21T08:41:13.026534+0000","last_peered":"2024-03-21T08:41:13.026534+0000","last_clean":"2024-03-21T08:41:13.026534+0000","last_became_active":"2024-03-21T08:41:12.334459+0000","last_became_peered":"2024-03-21T08:41:12.334459+0000","last_unstale":"2024-03-21T08:41:13.026534+0000","last_undegraded":"2024-03-21T08:41:13.026534+0000","last_fullsized":"2024-03-21T08:41:13.026534+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:17.097499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736898+0000","last_change":"2024-03-21T08:41:08.030635+0000","last_active":"2024-03-21T08:41:13.736898+0000","last_peered":"2024-03-21T08:41:13.736898+0000","last_clean":"2024-03-21T08:41:13.736898+0000","last_became_active":"2024-03-21T08:41:08.028635+0000","last_became_peered":"2024-03-21T08:41:08.028635+0000","last_unstale":"2024-03-21T08:41:13.736898+0000","last_undegraded":"2024-03-21T08:41:13.736898+0000","last_fullsized":"2024-03-21T08:41:13.736898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:14.013088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026041+0000","last_change":"2024-03-21T08:41:07.004839+0000","last_active":"2024-03-21T08:41:13.026041+0000","last_peered":"2024-03-21T08:41:13.026041+0000","last_clean":"2024-03-21T08:41:13.026041+0000","last_became_active":"2024-03-21T08:41:07.004409+0000","last_became_peered":"2024-03-21T08:41:07.004409+0000","last_unstale":"2024-03-21T08:41:13.026041+0000","last_undegraded":"2024-03-21T08:41:13.026041+0000","last_fullsized":"2024-03-21T08:41:13.026041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025597+0000","last_change":"2024-03-21T08:41:12.334998+0000","last_active":"2024-03-21T08:41:13.025597+0000","last_peered":"2024-03-21T08:41:13.025597+0000","last_clean":"2024-03-21T08:41:13.025597+0000","last_became_active":"2024-03-21T08:41:12.334610+0000","last_became_peered":"2024-03-21T08:41:12.334610+0000","last_unstale":"2024-03-21T08:41:13.025597+0000","last_undegraded":"2024-03-21T08:41:13.025597+0000","last_fullsized":"2024-03-21T08:41:13.025597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:04.540298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261002+0000","last_change":"2024-03-21T08:41:08.014545+0000","last_active":"2024-03-21T08:41:14.261002+0000","last_peered":"2024-03-21T08:41:14.261002+0000","last_clean":"2024-03-21T08:41:14.261002+0000","last_became_active":"2024-03-21T08:41:08.013815+0000","last_became_peered":"2024-03-21T08:41:08.013815+0000","last_unstale":"2024-03-21T08:41:14.261002+0000","last_undegraded":"2024-03-21T08:41:14.261002+0000","last_fullsized":"2024-03-21T08:41:14.261002+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:09:07.068892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263256+0000","last_change":"2024-03-21T08:41:07.011021+0000","last_active":"2024-03-21T08:41:14.263256+0000","last_peered":"2024-03-21T08:41:14.263256+0000","last_clean":"2024-03-21T08:41:14.263256+0000","last_became_active":"2024-03-21T08:41:07.010411+0000","last_became_peered":"2024-03-21T08:41:07.010411+0000","last_unstale":"2024-03-21T08:41:14.263256+0000","last_undegraded":"2024-03-21T08:41:14.263256+0000","last_fullsized":"2024-03-21T08:41:14.263256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.596104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161403+0000","last_change":"2024-03-21T08:41:12.055570+0000","last_active":"2024-03-21T08:41:13.161403+0000","last_peered":"2024-03-21T08:41:13.161403+0000","last_clean":"2024-03-21T08:41:13.161403+0000","last_became_active":"2024-03-21T08:41:12.055174+0000","last_became_peered":"2024-03-21T08:41:12.055174+0000","last_unstale":"2024-03-21T08:41:13.161403+0000","last_undegraded":"2024-03-21T08:41:13.161403+0000","last_fullsized":"2024-03-21T08:41:13.161403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:45:15.642543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737253+0000","last_change":"2024-03-21T08:41:08.037018+0000","last_active":"2024-03-21T08:41:13.737253+0000","last_peered":"2024-03-21T08:41:13.737253+0000","last_clean":"2024-03-21T08:41:13.737253+0000","last_became_active":"2024-03-21T08:41:08.036676+0000","last_became_peered":"2024-03-21T08:41:08.036676+0000","last_unstale":"2024-03-21T08:41:13.737253+0000","last_undegraded":"2024-03-21T08:41:13.737253+0000","last_fullsized":"2024-03-21T08:41:13.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:59:56.634930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025975+0000","last_change":"2024-03-21T08:41:07.004711+0000","last_active":"2024-03-21T08:41:13.025975+0000","last_peered":"2024-03-21T08:41:13.025975+0000","last_clean":"2024-03-21T08:41:13.025975+0000","last_became_active":"2024-03-21T08:41:07.004192+0000","last_became_peered":"2024-03-21T08:41:07.004192+0000","last_unstale":"2024-03-21T08:41:13.025975+0000","last_undegraded":"2024-03-21T08:41:13.025975+0000","last_fullsized":"2024-03-21T08:41:13.025975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260477+0000","last_change":"2024-03-21T08:41:12.341167+0000","last_active":"2024-03-21T08:41:14.260477+0000","last_peered":"2024-03-21T08:41:14.260477+0000","last_clean":"2024-03-21T08:41:14.260477+0000","last_became_active":"2024-03-21T08:41:12.339602+0000","last_became_peered":"2024-03-21T08:41:12.339602+0000","last_unstale":"2024-03-21T08:41:14.260477+0000","last_undegraded":"2024-03-21T08:41:14.260477+0000","last_fullsized":"2024-03-21T08:41:14.260477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:29:36.296868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737088+0000","last_change":"2024-03-21T08:41:08.033751+0000","last_active":"2024-03-21T08:41:13.737088+0000","last_peered":"2024-03-21T08:41:13.737088+0000","last_clean":"2024-03-21T08:41:13.737088+0000","last_became_active":"2024-03-21T08:41:08.033458+0000","last_became_peered":"2024-03-21T08:41:08.033458+0000","last_unstale":"2024-03-21T08:41:13.737088+0000","last_undegraded":"2024-03-21T08:41:13.737088+0000","last_fullsized":"2024-03-21T08:41:13.737088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:38:13.065681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027669+0000","last_change":"2024-03-21T08:41:07.017025+0000","last_active":"2024-03-21T08:41:13.027669+0000","last_peered":"2024-03-21T08:41:13.027669+0000","last_clean":"2024-03-21T08:41:13.027669+0000","last_became_active":"2024-03-21T08:41:07.016654+0000","last_became_peered":"2024-03-21T08:41:07.016654+0000","last_unstale":"2024-03-21T08:41:13.027669+0000","last_undegraded":"2024-03-21T08:41:13.027669+0000","last_fullsized":"2024-03-21T08:41:13.027669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:10:03.436446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025227+0000","last_change":"2024-03-21T08:41:12.048497+0000","last_active":"2024-03-21T08:41:13.025227+0000","last_peered":"2024-03-21T08:41:13.025227+0000","last_clean":"2024-03-21T08:41:13.025227+0000","last_became_active":"2024-03-21T08:41:12.047987+0000","last_became_peered":"2024-03-21T08:41:12.047987+0000","last_unstale":"2024-03-21T08:41:13.025227+0000","last_undegraded":"2024-03-21T08:41:13.025227+0000","last_fullsized":"2024-03-21T08:41:13.025227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:39.659897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736537+0000","last_change":"2024-03-21T08:41:08.033999+0000","last_active":"2024-03-21T08:41:13.736537+0000","last_peered":"2024-03-21T08:41:13.736537+0000","last_clean":"2024-03-21T08:41:13.736537+0000","last_became_active":"2024-03-21T08:41:08.033015+0000","last_became_peered":"2024-03-21T08:41:08.033015+0000","last_unstale":"2024-03-21T08:41:13.736537+0000","last_undegraded":"2024-03-21T08:41:13.736537+0000","last_fullsized":"2024-03-21T08:41:13.736537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:16:25.591202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025409+0000","last_change":"2024-03-21T08:41:07.001332+0000","last_active":"2024-03-21T08:41:13.025409+0000","last_peered":"2024-03-21T08:41:13.025409+0000","last_clean":"2024-03-21T08:41:13.025409+0000","last_became_active":"2024-03-21T08:41:06.999605+0000","last_became_peered":"2024-03-21T08:41:06.999605+0000","last_unstale":"2024-03-21T08:41:13.025409+0000","last_undegraded":"2024-03-21T08:41:13.025409+0000","last_fullsized":"2024-03-21T08:41:13.025409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:06:31.096538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161359+0000","last_change":"2024-03-21T08:41:12.059105+0000","last_active":"2024-03-21T08:41:13.161359+0000","last_peered":"2024-03-21T08:41:13.161359+0000","last_clean":"2024-03-21T08:41:13.161359+0000","last_became_active":"2024-03-21T08:41:12.058643+0000","last_became_peered":"2024-03-21T08:41:12.058643+0000","last_unstale":"2024-03-21T08:41:13.161359+0000","last_undegraded":"2024-03-21T08:41:13.161359+0000","last_fullsized":"2024-03-21T08:41:13.161359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:13:18.221689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025534+0000","last_change":"2024-03-21T08:41:08.010609+0000","last_active":"2024-03-21T08:41:13.025534+0000","last_peered":"2024-03-21T08:41:13.025534+0000","last_clean":"2024-03-21T08:41:13.025534+0000","last_became_active":"2024-03-21T08:41:08.010109+0000","last_became_peered":"2024-03-21T08:41:08.010109+0000","last_unstale":"2024-03-21T08:41:13.025534+0000","last_undegraded":"2024-03-21T08:41:13.025534+0000","last_fullsized":"2024-03-21T08:41:13.025534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:43.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026017+0000","last_change":"2024-03-21T08:41:07.010384+0000","last_active":"2024-03-21T08:41:13.026017+0000","last_peered":"2024-03-21T08:41:13.026017+0000","last_clean":"2024-03-21T08:41:13.026017+0000","last_became_active":"2024-03-21T08:41:07.010019+0000","last_became_peered":"2024-03-21T08:41:07.010019+0000","last_unstale":"2024-03-21T08:41:13.026017+0000","last_undegraded":"2024-03-21T08:41:13.026017+0000","last_fullsized":"2024-03-21T08:41:13.026017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:16.634005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262930+0000","last_change":"2024-03-21T08:41:12.341603+0000","last_active":"2024-03-21T08:41:14.262930+0000","last_peered":"2024-03-21T08:41:14.262930+0000","last_clean":"2024-03-21T08:41:14.262930+0000","last_became_active":"2024-03-21T08:41:12.341158+0000","last_became_peered":"2024-03-21T08:41:12.341158+0000","last_unstale":"2024-03-21T08:41:14.262930+0000","last_undegraded":"2024-03-21T08:41:14.262930+0000","last_fullsized":"2024-03-21T08:41:14.262930+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:02.246810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736756+0000","last_change":"2024-03-21T08:41:08.032505+0000","last_active":"2024-03-21T08:41:13.736756+0000","last_peered":"2024-03-21T08:41:13.736756+0000","last_clean":"2024-03-21T08:41:13.736756+0000","last_became_active":"2024-03-21T08:41:08.032026+0000","last_became_peered":"2024-03-21T08:41:08.032026+0000","last_unstale":"2024-03-21T08:41:13.736756+0000","last_undegraded":"2024-03-21T08:41:13.736756+0000","last_fullsized":"2024-03-21T08:41:13.736756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:21.306665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260844+0000","last_change":"2024-03-21T08:41:07.003264+0000","last_active":"2024-03-21T08:41:14.260844+0000","last_peered":"2024-03-21T08:41:14.260844+0000","last_clean":"2024-03-21T08:41:14.260844+0000","last_became_active":"2024-03-21T08:41:07.002969+0000","last_became_peered":"2024-03-21T08:41:07.002969+0000","last_unstale":"2024-03-21T08:41:14.260844+0000","last_undegraded":"2024-03-21T08:41:14.260844+0000","last_fullsized":"2024-03-21T08:41:14.260844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:19.390331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262423+0000","last_change":"2024-03-21T08:41:12.336949+0000","last_active":"2024-03-21T08:41:14.262423+0000","last_peered":"2024-03-21T08:41:14.262423+0000","last_clean":"2024-03-21T08:41:14.262423+0000","last_became_active":"2024-03-21T08:41:12.335654+0000","last_became_peered":"2024-03-21T08:41:12.335654+0000","last_unstale":"2024-03-21T08:41:14.262423+0000","last_undegraded":"2024-03-21T08:41:14.262423+0000","last_fullsized":"2024-03-21T08:41:14.262423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:54:40.677358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160969+0000","last_change":"2024-03-21T08:41:08.020852+0000","last_active":"2024-03-21T08:41:13.160969+0000","last_peered":"2024-03-21T08:41:13.160969+0000","last_clean":"2024-03-21T08:41:13.160969+0000","last_became_active":"2024-03-21T08:41:08.020547+0000","last_became_peered":"2024-03-21T08:41:08.020547+0000","last_unstale":"2024-03-21T08:41:13.160969+0000","last_undegraded":"2024-03-21T08:41:13.160969+0000","last_fullsized":"2024-03-21T08:41:13.160969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:08:56.518303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161459+0000","last_change":"2024-03-21T08:41:07.013979+0000","last_active":"2024-03-21T08:41:13.161459+0000","last_peered":"2024-03-21T08:41:13.161459+0000","last_clean":"2024-03-21T08:41:13.161459+0000","last_became_active":"2024-03-21T08:41:07.013725+0000","last_became_peered":"2024-03-21T08:41:07.013725+0000","last_unstale":"2024-03-21T08:41:13.161459+0000","last_undegraded":"2024-03-21T08:41:13.161459+0000","last_fullsized":"2024-03-21T08:41:13.161459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:03:13.978443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261111+0000","last_change":"2024-03-21T08:41:12.335008+0000","last_active":"2024-03-21T08:41:14.261111+0000","last_peered":"2024-03-21T08:41:14.261111+0000","last_clean":"2024-03-21T08:41:14.261111+0000","last_became_active":"2024-03-21T08:41:12.334635+0000","last_became_peered":"2024-03-21T08:41:12.334635+0000","last_unstale":"2024-03-21T08:41:14.261111+0000","last_undegraded":"2024-03-21T08:41:14.261111+0000","last_fullsized":"2024-03-21T08:41:14.261111+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736888+0000","last_change":"2024-03-21T08:41:08.026214+0000","last_active":"2024-03-21T08:41:13.736888+0000","last_peered":"2024-03-21T08:41:13.736888+0000","last_clean":"2024-03-21T08:41:13.736888+0000","last_became_active":"2024-03-21T08:41:08.025686+0000","last_became_peered":"2024-03-21T08:41:08.025686+0000","last_unstale":"2024-03-21T08:41:13.736888+0000","last_undegraded":"2024-03-21T08:41:13.736888+0000","last_fullsized":"2024-03-21T08:41:13.736888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:32:26.184390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:23.028503+0000","last_change":"2024-03-21T08:41:07.005099+0000","last_active":"2024-03-21T08:41:23.028503+0000","last_peered":"2024-03-21T08:41:23.028503+0000","last_clean":"2024-03-21T08:41:23.028503+0000","last_became_active":"2024-03-21T08:41:07.004825+0000","last_became_peered":"2024-03-21T08:41:07.004825+0000","last_unstale":"2024-03-21T08:41:23.028503+0000","last_undegraded":"2024-03-21T08:41:23.028503+0000","last_fullsized":"2024-03-21T08:41:23.028503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:27:05.173804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262224+0000","last_change":"2024-03-21T08:41:12.057213+0000","last_active":"2024-03-21T08:41:14.262224+0000","last_peered":"2024-03-21T08:41:14.262224+0000","last_clean":"2024-03-21T08:41:14.262224+0000","last_became_active":"2024-03-21T08:41:12.056539+0000","last_became_peered":"2024-03-21T08:41:12.056539+0000","last_unstale":"2024-03-21T08:41:14.262224+0000","last_undegraded":"2024-03-21T08:41:14.262224+0000","last_fullsized":"2024-03-21T08:41:14.262224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:35.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027129+0000","last_change":"2024-03-21T08:41:08.018576+0000","last_active":"2024-03-21T08:41:13.027129+0000","last_peered":"2024-03-21T08:41:13.027129+0000","last_clean":"2024-03-21T08:41:13.027129+0000","last_became_active":"2024-03-21T08:41:08.018277+0000","last_became_peered":"2024-03-21T08:41:08.018277+0000","last_unstale":"2024-03-21T08:41:13.027129+0000","last_undegraded":"2024-03-21T08:41:13.027129+0000","last_fullsized":"2024-03-21T08:41:13.027129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:00:26.730038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263165+0000","last_change":"2024-03-21T08:41:07.013500+0000","last_active":"2024-03-21T08:41:14.263165+0000","last_peered":"2024-03-21T08:41:14.263165+0000","last_clean":"2024-03-21T08:41:14.263165+0000","last_became_active":"2024-03-21T08:41:07.010794+0000","last_became_peered":"2024-03-21T08:41:07.010794+0000","last_unstale":"2024-03-21T08:41:14.263165+0000","last_undegraded":"2024-03-21T08:41:14.263165+0000","last_fullsized":"2024-03-21T08:41:14.263165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027024+0000","last_change":"2024-03-21T08:41:12.049156+0000","last_active":"2024-03-21T08:41:13.027024+0000","last_peered":"2024-03-21T08:41:13.027024+0000","last_clean":"2024-03-21T08:41:13.027024+0000","last_became_active":"2024-03-21T08:41:12.048737+0000","last_became_peered":"2024-03-21T08:41:12.048737+0000","last_unstale":"2024-03-21T08:41:13.027024+0000","last_undegraded":"2024-03-21T08:41:13.027024+0000","last_fullsized":"2024-03-21T08:41:13.027024+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:12.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161355+0000","last_change":"2024-03-21T08:41:08.012883+0000","last_active":"2024-03-21T08:41:13.161355+0000","last_peered":"2024-03-21T08:41:13.161355+0000","last_clean":"2024-03-21T08:41:13.161355+0000","last_became_active":"2024-03-21T08:41:08.012565+0000","last_became_peered":"2024-03-21T08:41:08.012565+0000","last_unstale":"2024-03-21T08:41:13.161355+0000","last_undegraded":"2024-03-21T08:41:13.161355+0000","last_fullsized":"2024-03-21T08:41:13.161355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261096+0000","last_change":"2024-03-21T08:41:07.013115+0000","last_active":"2024-03-21T08:41:14.261096+0000","last_peered":"2024-03-21T08:41:14.261096+0000","last_clean":"2024-03-21T08:41:14.261096+0000","last_became_active":"2024-03-21T08:41:07.012884+0000","last_became_peered":"2024-03-21T08:41:07.012884+0000","last_unstale":"2024-03-21T08:41:14.261096+0000","last_undegraded":"2024-03-21T08:41:14.261096+0000","last_fullsized":"2024-03-21T08:41:14.261096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:39.115443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261540+0000","last_change":"2024-03-21T08:41:12.051266+0000","last_active":"2024-03-21T08:41:14.261540+0000","last_peered":"2024-03-21T08:41:14.261540+0000","last_clean":"2024-03-21T08:41:14.261540+0000","last_became_active":"2024-03-21T08:41:12.050452+0000","last_became_peered":"2024-03-21T08:41:12.050452+0000","last_unstale":"2024-03-21T08:41:14.261540+0000","last_undegraded":"2024-03-21T08:41:14.261540+0000","last_fullsized":"2024-03-21T08:41:14.261540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:27:45.924757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261053+0000","last_change":"2024-03-21T08:41:08.029184+0000","last_active":"2024-03-21T08:41:14.261053+0000","last_peered":"2024-03-21T08:41:14.261053+0000","last_clean":"2024-03-21T08:41:14.261053+0000","last_became_active":"2024-03-21T08:41:08.028897+0000","last_became_peered":"2024-03-21T08:41:08.028897+0000","last_unstale":"2024-03-21T08:41:14.261053+0000","last_undegraded":"2024-03-21T08:41:14.261053+0000","last_fullsized":"2024-03-21T08:41:14.261053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:46:59.062390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026681+0000","last_change":"2024-03-21T08:41:07.021159+0000","last_active":"2024-03-21T08:41:13.026681+0000","last_peered":"2024-03-21T08:41:13.026681+0000","last_clean":"2024-03-21T08:41:13.026681+0000","last_became_active":"2024-03-21T08:41:07.020879+0000","last_became_peered":"2024-03-21T08:41:07.020879+0000","last_unstale":"2024-03-21T08:41:13.026681+0000","last_undegraded":"2024-03-21T08:41:13.026681+0000","last_fullsized":"2024-03-21T08:41:13.026681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:40:01.340794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261676+0000","last_change":"2024-03-21T08:41:12.340577+0000","last_active":"2024-03-21T08:41:14.261676+0000","last_peered":"2024-03-21T08:41:14.261676+0000","last_clean":"2024-03-21T08:41:14.261676+0000","last_became_active":"2024-03-21T08:41:12.339629+0000","last_became_peered":"2024-03-21T08:41:12.339629+0000","last_unstale":"2024-03-21T08:41:14.261676+0000","last_undegraded":"2024-03-21T08:41:14.261676+0000","last_fullsized":"2024-03-21T08:41:14.261676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:42.494428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161192+0000","last_change":"2024-03-21T08:41:12.340677+0000","last_active":"2024-03-21T08:41:13.161192+0000","last_peered":"2024-03-21T08:41:13.161192+0000","last_clean":"2024-03-21T08:41:13.161192+0000","last_became_active":"2024-03-21T08:41:12.340153+0000","last_became_peered":"2024-03-21T08:41:12.340153+0000","last_unstale":"2024-03-21T08:41:13.161192+0000","last_undegraded":"2024-03-21T08:41:13.161192+0000","last_fullsized":"2024-03-21T08:41:13.161192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:48.677726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261917+0000","last_change":"2024-03-21T08:41:08.004430+0000","last_active":"2024-03-21T08:41:14.261917+0000","last_peered":"2024-03-21T08:41:14.261917+0000","last_clean":"2024-03-21T08:41:14.261917+0000","last_became_active":"2024-03-21T08:41:08.003935+0000","last_became_peered":"2024-03-21T08:41:08.003935+0000","last_unstale":"2024-03-21T08:41:14.261917+0000","last_undegraded":"2024-03-21T08:41:14.261917+0000","last_fullsized":"2024-03-21T08:41:14.261917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260592+0000","last_change":"2024-03-21T08:41:07.011532+0000","last_active":"2024-03-21T08:41:14.260592+0000","last_peered":"2024-03-21T08:41:14.260592+0000","last_clean":"2024-03-21T08:41:14.260592+0000","last_became_active":"2024-03-21T08:41:07.008460+0000","last_became_peered":"2024-03-21T08:41:07.008460+0000","last_unstale":"2024-03-21T08:41:14.260592+0000","last_undegraded":"2024-03-21T08:41:14.260592+0000","last_fullsized":"2024-03-21T08:41:14.260592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026625+0000","last_change":"2024-03-21T08:41:12.054672+0000","last_active":"2024-03-21T08:41:13.026625+0000","last_peered":"2024-03-21T08:41:13.026625+0000","last_clean":"2024-03-21T08:41:13.026625+0000","last_became_active":"2024-03-21T08:41:12.054257+0000","last_became_peered":"2024-03-21T08:41:12.054257+0000","last_unstale":"2024-03-21T08:41:13.026625+0000","last_undegraded":"2024-03-21T08:41:13.026625+0000","last_fullsized":"2024-03-21T08:41:13.026625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:07:18.044906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261534+0000","last_change":"2024-03-21T08:41:08.015362+0000","last_active":"2024-03-21T08:41:14.261534+0000","last_peered":"2024-03-21T08:41:14.261534+0000","last_clean":"2024-03-21T08:41:14.261534+0000","last_became_active":"2024-03-21T08:41:08.015110+0000","last_became_peered":"2024-03-21T08:41:08.015110+0000","last_unstale":"2024-03-21T08:41:14.261534+0000","last_undegraded":"2024-03-21T08:41:14.261534+0000","last_fullsized":"2024-03-21T08:41:14.261534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261823+0000","last_change":"2024-03-21T08:41:07.001659+0000","last_active":"2024-03-21T08:41:14.261823+0000","last_peered":"2024-03-21T08:41:14.261823+0000","last_clean":"2024-03-21T08:41:14.261823+0000","last_became_active":"2024-03-21T08:41:07.001354+0000","last_became_peered":"2024-03-21T08:41:07.001354+0000","last_unstale":"2024-03-21T08:41:14.261823+0000","last_undegraded":"2024-03-21T08:41:14.261823+0000","last_fullsized":"2024-03-21T08:41:14.261823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:54:20.256597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026173+0000","last_change":"2024-03-21T08:41:07.009459+0000","last_active":"2024-03-21T08:41:13.026173+0000","last_peered":"2024-03-21T08:41:13.026173+0000","last_clean":"2024-03-21T08:41:13.026173+0000","last_became_active":"2024-03-21T08:41:07.009113+0000","last_became_peered":"2024-03-21T08:41:07.009113+0000","last_unstale":"2024-03-21T08:41:13.026173+0000","last_undegraded":"2024-03-21T08:41:13.026173+0000","last_fullsized":"2024-03-21T08:41:13.026173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:06.756017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161366+0000","last_change":"2024-03-21T08:41:07.996784+0000","last_active":"2024-03-21T08:41:13.161366+0000","last_peered":"2024-03-21T08:41:13.161366+0000","last_clean":"2024-03-21T08:41:13.161366+0000","last_became_active":"2024-03-21T08:41:07.996521+0000","last_became_peered":"2024-03-21T08:41:07.996521+0000","last_unstale":"2024-03-21T08:41:13.161366+0000","last_undegraded":"2024-03-21T08:41:13.161366+0000","last_fullsized":"2024-03-21T08:41:13.161366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025773+0000","last_change":"2024-03-21T08:41:12.052094+0000","last_active":"2024-03-21T08:41:13.025773+0000","last_peered":"2024-03-21T08:41:13.025773+0000","last_clean":"2024-03-21T08:41:13.025773+0000","last_became_active":"2024-03-21T08:41:12.051727+0000","last_became_peered":"2024-03-21T08:41:12.051727+0000","last_unstale":"2024-03-21T08:41:13.025773+0000","last_undegraded":"2024-03-21T08:41:13.025773+0000","last_fullsized":"2024-03-21T08:41:13.025773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:06:49.718295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262269+0000","last_change":"2024-03-21T08:41:07.012156+0000","last_active":"2024-03-21T08:41:14.262269+0000","last_peered":"2024-03-21T08:41:14.262269+0000","last_clean":"2024-03-21T08:41:14.262269+0000","last_became_active":"2024-03-21T08:41:07.011881+0000","last_became_peered":"2024-03-21T08:41:07.011881+0000","last_unstale":"2024-03-21T08:41:14.262269+0000","last_undegraded":"2024-03-21T08:41:14.262269+0000","last_fullsized":"2024-03-21T08:41:14.262269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:05.971941+0000","last_clean_scrub_stamp":"2024-03-21T08:41:05.971941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:04.222176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027548+0000","last_change":"2024-03-21T08:41:08.002324+0000","last_active":"2024-03-21T08:41:13.027548+0000","last_peered":"2024-03-21T08:41:13.027548+0000","last_clean":"2024-03-21T08:41:13.027548+0000","last_became_active":"2024-03-21T08:41:08.002038+0000","last_became_peered":"2024-03-21T08:41:08.002038+0000","last_unstale":"2024-03-21T08:41:13.027548+0000","last_undegraded":"2024-03-21T08:41:13.027548+0000","last_fullsized":"2024-03-21T08:41:13.027548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:20.396047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262336+0000","last_change":"2024-03-21T08:41:12.058878+0000","last_active":"2024-03-21T08:41:14.262336+0000","last_peered":"2024-03-21T08:41:14.262336+0000","last_clean":"2024-03-21T08:41:14.262336+0000","last_became_active":"2024-03-21T08:41:12.058446+0000","last_became_peered":"2024-03-21T08:41:12.058446+0000","last_unstale":"2024-03-21T08:41:14.262336+0000","last_undegraded":"2024-03-21T08:41:14.262336+0000","last_fullsized":"2024-03-21T08:41:14.262336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:11.009474+0000","last_clean_scrub_stamp":"2024-03-21T08:41:11.009474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:59:29.179384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025278+0000","last_change":"2024-03-21T08:41:08.010406+0000","last_active":"2024-03-21T08:41:13.025278+0000","last_peered":"2024-03-21T08:41:13.025278+0000","last_clean":"2024-03-21T08:41:13.025278+0000","last_became_active":"2024-03-21T08:41:08.009763+0000","last_became_peered":"2024-03-21T08:41:08.009763+0000","last_unstale":"2024-03-21T08:41:13.025278+0000","last_undegraded":"2024-03-21T08:41:13.025278+0000","last_fullsized":"2024-03-21T08:41:13.025278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025559+0000","last_change":"2024-03-21T08:41:08.009958+0000","last_active":"2024-03-21T08:41:13.025559+0000","last_peered":"2024-03-21T08:41:13.025559+0000","last_clean":"2024-03-21T08:41:13.025559+0000","last_became_active":"2024-03-21T08:41:08.009691+0000","last_became_peered":"2024-03-21T08:41:08.009691+0000","last_unstale":"2024-03-21T08:41:13.025559+0000","last_undegraded":"2024-03-21T08:41:13.025559+0000","last_fullsized":"2024-03-21T08:41:13.025559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:46.141146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260744+0000","last_change":"2024-03-21T08:41:08.004794+0000","last_active":"2024-03-21T08:41:14.260744+0000","last_peered":"2024-03-21T08:41:14.260744+0000","last_clean":"2024-03-21T08:41:14.260744+0000","last_became_active":"2024-03-21T08:41:08.003470+0000","last_became_peered":"2024-03-21T08:41:08.003470+0000","last_unstale":"2024-03-21T08:41:14.260744+0000","last_undegraded":"2024-03-21T08:41:14.260744+0000","last_fullsized":"2024-03-21T08:41:14.260744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161263+0000","last_change":"2024-03-21T08:41:08.014560+0000","last_active":"2024-03-21T08:41:13.161263+0000","last_peered":"2024-03-21T08:41:13.161263+0000","last_clean":"2024-03-21T08:41:13.161263+0000","last_became_active":"2024-03-21T08:41:08.014307+0000","last_became_peered":"2024-03-21T08:41:08.014307+0000","last_unstale":"2024-03-21T08:41:13.161263+0000","last_undegraded":"2024-03-21T08:41:13.161263+0000","last_fullsized":"2024-03-21T08:41:13.161263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:42.560268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262543+0000","last_change":"2024-03-21T08:41:08.026095+0000","last_active":"2024-03-21T08:41:14.262543+0000","last_peered":"2024-03-21T08:41:14.262543+0000","last_clean":"2024-03-21T08:41:14.262543+0000","last_became_active":"2024-03-21T08:41:08.025719+0000","last_became_peered":"2024-03-21T08:41:08.025719+0000","last_unstale":"2024-03-21T08:41:14.262543+0000","last_undegraded":"2024-03-21T08:41:14.262543+0000","last_fullsized":"2024-03-21T08:41:14.262543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:27:59.288655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262685+0000","last_change":"2024-03-21T08:41:08.007171+0000","last_active":"2024-03-21T08:41:14.262685+0000","last_peered":"2024-03-21T08:41:14.262685+0000","last_clean":"2024-03-21T08:41:14.262685+0000","last_became_active":"2024-03-21T08:41:08.006702+0000","last_became_peered":"2024-03-21T08:41:08.006702+0000","last_unstale":"2024-03-21T08:41:14.262685+0000","last_undegraded":"2024-03-21T08:41:14.262685+0000","last_fullsized":"2024-03-21T08:41:14.262685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026846+0000","last_change":"2024-03-21T08:41:08.029843+0000","last_active":"2024-03-21T08:41:13.026846+0000","last_peered":"2024-03-21T08:41:13.026846+0000","last_clean":"2024-03-21T08:41:13.026846+0000","last_became_active":"2024-03-21T08:41:08.029528+0000","last_became_peered":"2024-03-21T08:41:08.029528+0000","last_unstale":"2024-03-21T08:41:13.026846+0000","last_undegraded":"2024-03-21T08:41:13.026846+0000","last_fullsized":"2024-03-21T08:41:13.026846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262888+0000","last_change":"2024-03-21T08:41:08.013879+0000","last_active":"2024-03-21T08:41:14.262888+0000","last_peered":"2024-03-21T08:41:14.262888+0000","last_clean":"2024-03-21T08:41:14.262888+0000","last_became_active":"2024-03-21T08:41:08.013610+0000","last_became_peered":"2024-03-21T08:41:08.013610+0000","last_unstale":"2024-03-21T08:41:14.262888+0000","last_undegraded":"2024-03-21T08:41:14.262888+0000","last_fullsized":"2024-03-21T08:41:14.262888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260551+0000","last_change":"2024-03-21T08:41:08.001171+0000","last_active":"2024-03-21T08:41:14.260551+0000","last_peered":"2024-03-21T08:41:14.260551+0000","last_clean":"2024-03-21T08:41:14.260551+0000","last_became_active":"2024-03-21T08:41:08.000833+0000","last_became_peered":"2024-03-21T08:41:08.000833+0000","last_unstale":"2024-03-21T08:41:14.260551+0000","last_undegraded":"2024-03-21T08:41:14.260551+0000","last_fullsized":"2024-03-21T08:41:14.260551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:40:35.628079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261730+0000","last_change":"2024-03-21T08:41:08.014405+0000","last_active":"2024-03-21T08:41:14.261730+0000","last_peered":"2024-03-21T08:41:14.261730+0000","last_clean":"2024-03-21T08:41:14.261730+0000","last_became_active":"2024-03-21T08:41:08.014151+0000","last_became_peered":"2024-03-21T08:41:08.014151+0000","last_unstale":"2024-03-21T08:41:14.261730+0000","last_undegraded":"2024-03-21T08:41:14.261730+0000","last_fullsized":"2024-03-21T08:41:14.261730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260722+0000","last_change":"2024-03-21T08:41:08.019319+0000","last_active":"2024-03-21T08:41:14.260722+0000","last_peered":"2024-03-21T08:41:14.260722+0000","last_clean":"2024-03-21T08:41:14.260722+0000","last_became_active":"2024-03-21T08:41:08.019027+0000","last_became_peered":"2024-03-21T08:41:08.019027+0000","last_unstale":"2024-03-21T08:41:14.260722+0000","last_undegraded":"2024-03-21T08:41:14.260722+0000","last_fullsized":"2024-03-21T08:41:14.260722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:07:08.879531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161113+0000","last_change":"2024-03-21T08:41:08.018077+0000","last_active":"2024-03-21T08:41:13.161113+0000","last_peered":"2024-03-21T08:41:13.161113+0000","last_clean":"2024-03-21T08:41:13.161113+0000","last_became_active":"2024-03-21T08:41:08.017564+0000","last_became_peered":"2024-03-21T08:41:08.017564+0000","last_unstale":"2024-03-21T08:41:13.161113+0000","last_undegraded":"2024-03-21T08:41:13.161113+0000","last_fullsized":"2024-03-21T08:41:13.161113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:17.581392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026399+0000","last_change":"2024-03-21T08:41:08.018108+0000","last_active":"2024-03-21T08:41:13.026399+0000","last_peered":"2024-03-21T08:41:13.026399+0000","last_clean":"2024-03-21T08:41:13.026399+0000","last_became_active":"2024-03-21T08:41:08.017766+0000","last_became_peered":"2024-03-21T08:41:08.017766+0000","last_unstale":"2024-03-21T08:41:13.026399+0000","last_undegraded":"2024-03-21T08:41:13.026399+0000","last_fullsized":"2024-03-21T08:41:13.026399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:35.105057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737069+0000","last_change":"2024-03-21T08:41:08.021075+0000","last_active":"2024-03-21T08:41:13.737069+0000","last_peered":"2024-03-21T08:41:13.737069+0000","last_clean":"2024-03-21T08:41:13.737069+0000","last_became_active":"2024-03-21T08:41:08.020798+0000","last_became_peered":"2024-03-21T08:41:08.020798+0000","last_unstale":"2024-03-21T08:41:13.737069+0000","last_undegraded":"2024-03-21T08:41:13.737069+0000","last_fullsized":"2024-03-21T08:41:13.737069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:51:06.452855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.025886+0000","last_change":"2024-03-21T08:41:08.008591+0000","last_active":"2024-03-21T08:41:13.025886+0000","last_peered":"2024-03-21T08:41:13.025886+0000","last_clean":"2024-03-21T08:41:13.025886+0000","last_became_active":"2024-03-21T08:41:08.008328+0000","last_became_peered":"2024-03-21T08:41:08.008328+0000","last_unstale":"2024-03-21T08:41:13.025886+0000","last_undegraded":"2024-03-21T08:41:13.025886+0000","last_fullsized":"2024-03-21T08:41:13.025886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:23:00.350687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261256+0000","last_change":"2024-03-21T08:41:08.023131+0000","last_active":"2024-03-21T08:41:14.261256+0000","last_peered":"2024-03-21T08:41:14.261256+0000","last_clean":"2024-03-21T08:41:14.261256+0000","last_became_active":"2024-03-21T08:41:08.022844+0000","last_became_peered":"2024-03-21T08:41:08.022844+0000","last_unstale":"2024-03-21T08:41:14.261256+0000","last_undegraded":"2024-03-21T08:41:14.261256+0000","last_fullsized":"2024-03-21T08:41:14.261256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:10:44.645908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260479+0000","last_change":"2024-03-21T08:41:08.008995+0000","last_active":"2024-03-21T08:41:14.260479+0000","last_peered":"2024-03-21T08:41:14.260479+0000","last_clean":"2024-03-21T08:41:14.260479+0000","last_became_active":"2024-03-21T08:41:08.008707+0000","last_became_peered":"2024-03-21T08:41:08.008707+0000","last_unstale":"2024-03-21T08:41:14.260479+0000","last_undegraded":"2024-03-21T08:41:14.260479+0000","last_fullsized":"2024-03-21T08:41:14.260479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:49:11.625944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261974+0000","last_change":"2024-03-21T08:41:08.007429+0000","last_active":"2024-03-21T08:41:14.261974+0000","last_peered":"2024-03-21T08:41:14.261974+0000","last_clean":"2024-03-21T08:41:14.261974+0000","last_became_active":"2024-03-21T08:41:08.007118+0000","last_became_peered":"2024-03-21T08:41:08.007118+0000","last_unstale":"2024-03-21T08:41:14.261974+0000","last_undegraded":"2024-03-21T08:41:14.261974+0000","last_fullsized":"2024-03-21T08:41:14.261974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:25:32.914939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.160940+0000","last_change":"2024-03-21T08:41:08.002687+0000","last_active":"2024-03-21T08:41:13.160940+0000","last_peered":"2024-03-21T08:41:13.160940+0000","last_clean":"2024-03-21T08:41:13.160940+0000","last_became_active":"2024-03-21T08:41:08.002397+0000","last_became_peered":"2024-03-21T08:41:08.002397+0000","last_unstale":"2024-03-21T08:41:13.160940+0000","last_undegraded":"2024-03-21T08:41:13.160940+0000","last_fullsized":"2024-03-21T08:41:13.160940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:12:01.877634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.262084+0000","last_change":"2024-03-21T08:41:08.011489+0000","last_active":"2024-03-21T08:41:14.262084+0000","last_peered":"2024-03-21T08:41:14.262084+0000","last_clean":"2024-03-21T08:41:14.262084+0000","last_became_active":"2024-03-21T08:41:08.011227+0000","last_became_peered":"2024-03-21T08:41:08.011227+0000","last_unstale":"2024-03-21T08:41:14.262084+0000","last_undegraded":"2024-03-21T08:41:14.262084+0000","last_fullsized":"2024-03-21T08:41:14.262084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:14:50.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.027108+0000","last_change":"2024-03-21T08:41:08.010854+0000","last_active":"2024-03-21T08:41:13.027108+0000","last_peered":"2024-03-21T08:41:13.027108+0000","last_clean":"2024-03-21T08:41:13.027108+0000","last_became_active":"2024-03-21T08:41:08.010557+0000","last_became_peered":"2024-03-21T08:41:08.010557+0000","last_unstale":"2024-03-21T08:41:13.027108+0000","last_undegraded":"2024-03-21T08:41:13.027108+0000","last_fullsized":"2024-03-21T08:41:13.027108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:01.730277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026506+0000","last_change":"2024-03-21T08:41:08.014812+0000","last_active":"2024-03-21T08:41:13.026506+0000","last_peered":"2024-03-21T08:41:13.026506+0000","last_clean":"2024-03-21T08:41:13.026506+0000","last_became_active":"2024-03-21T08:41:08.014508+0000","last_became_peered":"2024-03-21T08:41:08.014508+0000","last_unstale":"2024-03-21T08:41:13.026506+0000","last_undegraded":"2024-03-21T08:41:13.026506+0000","last_fullsized":"2024-03-21T08:41:13.026506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:09:32.520322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.026766+0000","last_change":"2024-03-21T08:41:08.013816+0000","last_active":"2024-03-21T08:41:13.026766+0000","last_peered":"2024-03-21T08:41:13.026766+0000","last_clean":"2024-03-21T08:41:13.026766+0000","last_became_active":"2024-03-21T08:41:08.013585+0000","last_became_peered":"2024-03-21T08:41:08.013585+0000","last_unstale":"2024-03-21T08:41:13.026766+0000","last_undegraded":"2024-03-21T08:41:13.026766+0000","last_fullsized":"2024-03-21T08:41:13.026766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:53:24.320760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737247+0000","last_change":"2024-03-21T08:41:08.031989+0000","last_active":"2024-03-21T08:41:13.737247+0000","last_peered":"2024-03-21T08:41:13.737247+0000","last_clean":"2024-03-21T08:41:13.737247+0000","last_became_active":"2024-03-21T08:41:08.031755+0000","last_became_peered":"2024-03-21T08:41:08.031755+0000","last_unstale":"2024-03-21T08:41:13.737247+0000","last_undegraded":"2024-03-21T08:41:13.737247+0000","last_fullsized":"2024-03-21T08:41:13.737247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:37.172205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.261442+0000","last_change":"2024-03-21T08:41:08.015070+0000","last_active":"2024-03-21T08:41:14.261442+0000","last_peered":"2024-03-21T08:41:14.261442+0000","last_clean":"2024-03-21T08:41:14.261442+0000","last_became_active":"2024-03-21T08:41:08.012083+0000","last_became_peered":"2024-03-21T08:41:08.012083+0000","last_unstale":"2024-03-21T08:41:14.261442+0000","last_undegraded":"2024-03-21T08:41:14.261442+0000","last_fullsized":"2024-03-21T08:41:14.261442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:56.646542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736535+0000","last_change":"2024-03-21T08:41:08.020740+0000","last_active":"2024-03-21T08:41:13.736535+0000","last_peered":"2024-03-21T08:41:13.736535+0000","last_clean":"2024-03-21T08:41:13.736535+0000","last_became_active":"2024-03-21T08:41:08.020396+0000","last_became_peered":"2024-03-21T08:41:08.020396+0000","last_unstale":"2024-03-21T08:41:13.736535+0000","last_undegraded":"2024-03-21T08:41:13.736535+0000","last_fullsized":"2024-03-21T08:41:13.736535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:59:37.815273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260654+0000","last_change":"2024-03-21T08:41:08.010314+0000","last_active":"2024-03-21T08:41:14.260654+0000","last_peered":"2024-03-21T08:41:14.260654+0000","last_clean":"2024-03-21T08:41:14.260654+0000","last_became_active":"2024-03-21T08:41:08.009693+0000","last_became_peered":"2024-03-21T08:41:08.009693+0000","last_unstale":"2024-03-21T08:41:14.260654+0000","last_undegraded":"2024-03-21T08:41:14.260654+0000","last_fullsized":"2024-03-21T08:41:14.260654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:48:06.903220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.736584+0000","last_change":"2024-03-21T08:41:08.020466+0000","last_active":"2024-03-21T08:41:13.736584+0000","last_peered":"2024-03-21T08:41:13.736584+0000","last_clean":"2024-03-21T08:41:13.736584+0000","last_became_active":"2024-03-21T08:41:08.020169+0000","last_became_peered":"2024-03-21T08:41:08.020169+0000","last_unstale":"2024-03-21T08:41:13.736584+0000","last_undegraded":"2024-03-21T08:41:13.736584+0000","last_fullsized":"2024-03-21T08:41:13.736584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:55:00.508480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.260900+0000","last_change":"2024-03-21T08:41:08.014151+0000","last_active":"2024-03-21T08:41:14.260900+0000","last_peered":"2024-03-21T08:41:14.260900+0000","last_clean":"2024-03-21T08:41:14.260900+0000","last_became_active":"2024-03-21T08:41:08.013319+0000","last_became_peered":"2024-03-21T08:41:08.013319+0000","last_unstale":"2024-03-21T08:41:14.260900+0000","last_undegraded":"2024-03-21T08:41:14.260900+0000","last_fullsized":"2024-03-21T08:41:14.260900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:44:16.725366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.161103+0000","last_change":"2024-03-21T08:41:08.002931+0000","last_active":"2024-03-21T08:41:13.161103+0000","last_peered":"2024-03-21T08:41:13.161103+0000","last_clean":"2024-03-21T08:41:13.161103+0000","last_became_active":"2024-03-21T08:41:08.001953+0000","last_became_peered":"2024-03-21T08:41:08.001953+0000","last_unstale":"2024-03-21T08:41:13.161103+0000","last_undegraded":"2024-03-21T08:41:13.161103+0000","last_fullsized":"2024-03-21T08:41:13.161103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:28:16.190101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:14.263385+0000","last_change":"2024-03-21T08:41:08.005883+0000","last_active":"2024-03-21T08:41:14.263385+0000","last_peered":"2024-03-21T08:41:14.263385+0000","last_clean":"2024-03-21T08:41:14.263385+0000","last_became_active":"2024-03-21T08:41:08.005503+0000","last_became_peered":"2024-03-21T08:41:08.005503+0000","last_unstale":"2024-03-21T08:41:14.263385+0000","last_undegraded":"2024-03-21T08:41:14.263385+0000","last_fullsized":"2024-03-21T08:41:14.263385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:33:22.969177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-21T08:41:13.737138+0000","last_change":"2024-03-21T08:41:08.030286+0000","last_active":"2024-03-21T08:41:13.737138+0000","last_peered":"2024-03-21T08:41:13.737138+0000","last_clean":"2024-03-21T08:41:13.737138+0000","last_became_active":"2024-03-21T08:41:08.029830+0000","last_became_peered":"2024-03-21T08:41:08.029830+0000","last_unstale":"2024-03-21T08:41:13.737138+0000","last_undegraded":"2024-03-21T08:41:13.737138+0000","last_fullsized":"2024-03-21T08:41:13.737138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:41:06.977656+0000","last_clean_scrub_stamp":"2024-03-21T08:41:06.977656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:34:31.510333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509451,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":693167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":102377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":106059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":693223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:41:32.294 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-21T08:41:32.294 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T08:41:32.294 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-21T08:41:32.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-21T08:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.469+0000 7fb1c9a00640 1 Processor -- start 2024-03-21T08:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.469+0000 7fb1c9a00640 1 -- start start 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1c40cf500 0x7fb1c40ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c40cc1f0 con 0x7fb1c415ba20 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c415be00 con 0x7fb1c406b880 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c40cfa00 con 0x7fb1c40cf500 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1c40cf500 0x7fb1c40ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1c40cf500 0x7fb1c40ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55658/0 (socket says 172.21.15.28:55658) 2024-03-21T08:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 -- 172.21.15.28:0/1226425099 learned_addr learned my addr 172.21.15.28:0/1226425099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c3e00640 1 --1- 172.21.15.28:0/1226425099 >> v1:172.21.15.28:6789/0 conn(0x7fb1c415ba20 0x7fb1c406b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38614/0 (socket says 172.21.15.28:38614) 2024-03-21T08:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 -- 172.21.15.28:0/1226425099 >> v1:172.21.15.151:6789/0 conn(0x7fb1c406b880 legacy=0x7fb1c40cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 -- 172.21.15.28:0/1226425099 >> v1:172.21.15.28:6789/0 conn(0x7fb1c415ba20 legacy=0x7fb1c406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.473+0000 7fb1c9000640 1 -- 172.21.15.28:0/1226425099 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40cfce0 con 0x7fb1c40cf500 2024-03-21T08:41:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9000640 1 --2- 172.21.15.28:0/1226425099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c40cf500 0x7fb1c40ccdc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009e80 tx=0x7fb1c40cdb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ca2d3e77996d23 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3400640 1 -- 172.21.15.28:0/1226425099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0a58a0 con 0x7fb1c40cf500 2024-03-21T08:41:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3400640 1 -- 172.21.15.28:0/1226425099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc0a5a40 con 0x7fb1c40cf500 2024-03-21T08:41:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3400640 1 -- 172.21.15.28:0/1226425099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0a5e10 con 0x7fb1c40cf500 2024-03-21T08:41:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- 172.21.15.28:0/1226425099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c40cf500 msgr2=0x7fb1c40ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/1226425099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c40cf500 0x7fb1c40ccdc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009e80 tx=0x7fb1c40cdb10 comp rx=0 tx=0).stop 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- 172.21.15.28:0/1226425099 shutdown_connections 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/1226425099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c40cf500 0x7fb1c40ccdc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- 172.21.15.28:0/1226425099 >> 172.21.15.28:0/1226425099 conn(0x7fb1c40b2280 msgr2=0x7fb1c40b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- 172.21.15.28:0/1226425099 shutdown_connections 2024-03-21T08:41:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- 172.21.15.28:0/1226425099 wait complete. 2024-03-21T08:41:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 Processor -- start 2024-03-21T08:41:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- start start 2024-03-21T08:41:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 0x7fb1c415e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1c415ba20 0x7fb1c4154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c4169840 con 0x7fb1c406b880 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c41696c0 con 0x7fb1c40cf500 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c41699c0 con 0x7fb1c415ba20 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55674/0 (socket says 172.21.15.28:55674) 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c3e00640 1 -- 172.21.15.28:0/3316441874 learned_addr learned my addr 172.21.15.28:0/3316441874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1bbe00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 0x7fb1c415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.477+0000 7fb1c9000640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1c415ba20 0x7fb1c4154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 -- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1c415ba20 msgr2=0x7fb1c4154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1c415ba20 0x7fb1c4154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 -- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 msgr2=0x7fb1c415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 0x7fb1c415e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 -- 172.21.15.28:0/3316441874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c415d5c0 con 0x7fb1c406b880 2024-03-21T08:41:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1bbe00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 0x7fb1c415e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c3e00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400c420 tx=0x7fb1b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b40074e0 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b4007680 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4004a30 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c40c98f0 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c40c9eb0 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c406d210 con 0x7fb1c406b880 2024-03-21T08:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb1b401a020 con 0x7fb1c406b880 2024-03-21T08:41:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 0x7fb1a0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb1b4085a60 con 0x7fb1c406b880 2024-03-21T08:41:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.481+0000 7fb1bbe00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 0x7fb1a0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.485+0000 7fb1bbe00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 0x7fb1a0069fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40c76c0 tx=0x7fb1ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.485+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb1c40cc1f0 con 0x7fb1c406b880 2024-03-21T08:41:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb1c4075270 con 0x7fb1c406b880 2024-03-21T08:41:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c0c00640 1 -- 172.21.15.28:0/3316441874 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb1b4056090 con 0x7fb1c406b880 2024-03-21T08:41:32.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:32.688 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T08:41:32.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 msgr2=0x7fb1a0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 0x7fb1a0069fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40c76c0 tx=0x7fb1ac009040 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 msgr2=0x7fb1c4166820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400c420 tx=0x7fb1b400c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 shutdown_connections 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1a0067b20 0x7fb1a0069fe0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1c415ba20 0x7fb1c4154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1c40cf500 0x7fb1c415e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 --2- 172.21.15.28:0/3316441874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c406b880 0x7fb1c4166820 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 >> 172.21.15.28:0/3316441874 conn(0x7fb1c40b2280 msgr2=0x7fb1c4070220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 shutdown_connections 2024-03-21T08:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.685+0000 7fb1c9a00640 1 -- 172.21.15.28:0/3316441874 wait complete. 2024-03-21T08:41:32.710 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-21T08:41:32.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.845+0000 7f882ac00640 1 Processor -- start 2024-03-21T08:41:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.845+0000 7f882ac00640 1 -- start start 2024-03-21T08:41:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f882ac00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 0x7f882415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f882ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 0x7f88240ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f882ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882415bf00 con 0x7f8824071980 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f882ac00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88240cedb0 con 0x7f88240ce490 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f882ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882415ebf0 con 0x7f882415ba70 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f8828c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8824071980 0x7f8824071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38630/0 (socket says 172.21.15.28:38630) 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f8828c00640 1 -- 172.21.15.28:0/3125679632 learned_addr learned my addr 172.21.15.28:0/3125679632 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f881be00640 1 --2- 172.21.15.28:0/3125679632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 0x7f88240ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f8823e00640 1 --2- 172.21.15.28:0/3125679632 >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 0x7f882415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f881be00640 1 -- 172.21.15.28:0/3125679632 >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 msgr2=0x7f882415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f8823400640 1 -- 172.21.15.28:0/3125679632 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942277349 0 0) 0x7f882415bf00 con 0x7f8824071980 2024-03-21T08:41:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f881be00640 1 --2- 172.21.15.28:0/3125679632 >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 0x7f882415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f881be00640 1 -- 172.21.15.28:0/3125679632 >> v1:172.21.15.28:6789/0 conn(0x7f8824071980 legacy=0x7f8824071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f881be00640 1 -- 172.21.15.28:0/3125679632 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882415eed0 con 0x7f882415ba70 2024-03-21T08:41:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.849+0000 7f8823e00640 1 --2- 172.21.15.28:0/3125679632 >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 0x7f882415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f881be00640 1 --2- 172.21.15.28:0/3125679632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 0x7f88240ce870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f880c0097c0 tx=0x7f880c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22e19d9488db3181 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f8823400640 1 -- 172.21.15.28:0/3125679632 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c0b0070 con 0x7f882415ba70 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f8823400640 1 -- 172.21.15.28:0/3125679632 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c0ab600 con 0x7f882415ba70 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f8823400640 1 -- 172.21.15.28:0/3125679632 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c0ab9d0 con 0x7f882415ba70 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- 172.21.15.28:0/3125679632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 msgr2=0x7f88240ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 --2- 172.21.15.28:0/3125679632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 0x7f88240ce870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f880c0097c0 tx=0x7f880c01c410 comp rx=0 tx=0).stop 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- 172.21.15.28:0/3125679632 shutdown_connections 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 --2- 172.21.15.28:0/3125679632 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f882415ba70 0x7f88240ce870 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 --2- 172.21.15.28:0/3125679632 >> v2:172.21.15.151:3300/0 conn(0x7f88240ce490 0x7f882415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- 172.21.15.28:0/3125679632 >> 172.21.15.28:0/3125679632 conn(0x7f88240cc230 msgr2=0x7f88240cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- 172.21.15.28:0/3125679632 shutdown_connections 2024-03-21T08:41:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- 172.21.15.28:0/3125679632 wait complete. 2024-03-21T08:41:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 Processor -- start 2024-03-21T08:41:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.853+0000 7f882ac00640 1 -- start start 2024-03-21T08:41:32.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8824071980 0x7f882414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f882415ba70 0x7f882415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241681c0 con 0x7f88240ce490 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824155390 con 0x7f882415ba70 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8824168340 con 0x7f8824071980 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55680/0 (socket says 172.21.15.28:55680) 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 -- 172.21.15.28:0/2419302758 learned_addr learned my addr 172.21.15.28:0/2419302758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8823e00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8824071980 0x7f882414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8828c00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f882415ba70 0x7f882415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 -- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8824071980 msgr2=0x7f882414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8824071980 0x7f882414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 -- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f882415ba70 msgr2=0x7f882415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f882415ba70 0x7f882415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 -- 172.21.15.28:0/2419302758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882415c3a0 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f881be00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f880c01a040 tx=0x7f880c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c0ba020 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c0b5460 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c0b0040 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882415c680 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882415cc40 con 0x7f88240ce490 2024-03-21T08:41:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882415c130 con 0x7f88240ce490 2024-03-21T08:41:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.857+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f880c0bd020 con 0x7f88240ce490 2024-03-21T08:41:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.861+0000 7f8820c00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 0x7f880006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.861+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f880c120360 con 0x7f88240ce490 2024-03-21T08:41:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.861+0000 7f8823e00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 0x7f880006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.861+0000 7f8823e00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 0x7f880006a010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8814009580 tx=0x7f8814009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:32.865+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f880c0c4030 con 0x7f88240ce490 2024-03-21T08:41:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.025+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f882404f410 con 0x7f88240ce490 2024-03-21T08:41:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f8820c00640 1 -- 172.21.15.28:0/2419302758 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f880c016090 con 0x7f88240ce490 2024-03-21T08:41:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:33.032 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:33.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 msgr2=0x7f880006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 0x7f880006a010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8814009580 tx=0x7f8814009190 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 msgr2=0x7f882414c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f880c01a040 tx=0x7f880c0191b0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 shutdown_connections 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f882415ba70 0x7f882415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88240ce490 0x7f882414c800 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8800067b50 0x7f880006a010 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 --2- 172.21.15.28:0/2419302758 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8824071980 0x7f882414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 >> 172.21.15.28:0/2419302758 conn(0x7f88240cc230 msgr2=0x7f882415e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 shutdown_connections 2024-03-21T08:41:33.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.029+0000 7f882ac00640 1 -- 172.21.15.28:0/2419302758 wait complete. 2024-03-21T08:41:33.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:33.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.193+0000 7f36fa200640 1 Processor -- start 2024-03-21T08:41:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.193+0000 7f36fa200640 1 -- start start 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36fa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f41442b0 0x7f36f40b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36fa200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f40b2d20 con 0x7f36f4144690 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36fa200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f40b3000 con 0x7f36f4071a80 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36fa200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f40b2ad0 con 0x7f36f41442b0 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f41442b0 0x7f36f40b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36eaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f36f4144690 0x7f36f40aebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38642/0 (socket says 172.21.15.28:38642) 2024-03-21T08:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f41442b0 0x7f36f40b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55684/0 (socket says 172.21.15.28:55684) 2024-03-21T08:41:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36eaa00640 1 -- 172.21.15.28:0/1827866519 learned_addr learned my addr 172.21.15.28:0/1827866519 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f3400640 1 -- 172.21.15.28:0/1827866519 >> v1:172.21.15.151:6789/0 conn(0x7f36f4071a80 legacy=0x7f36f4071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f3400640 1 -- 172.21.15.28:0/1827866519 >> v1:172.21.15.28:6789/0 conn(0x7f36f4144690 legacy=0x7f36f40aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f3400640 1 -- 172.21.15.28:0/1827866519 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f40b32e0 con 0x7f36f41442b0 2024-03-21T08:41:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.197+0000 7f36f2a00640 1 -- 172.21.15.28:0/1827866519 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056522927 0 0) 0x7f36f40b2d20 con 0x7f36f4144690 2024-03-21T08:41:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36f3400640 1 --2- 172.21.15.28:0/1827866519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f40b3dc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009d90 tx=0x7f36e401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1f67fb534d858a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36f2a00640 1 -- 172.21.15.28:0/1827866519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e40b2070 con 0x7f36f41442b0 2024-03-21T08:41:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36f2a00640 1 -- 172.21.15.28:0/1827866519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e40a5a00 con 0x7f36f41442b0 2024-03-21T08:41:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36f2a00640 1 -- 172.21.15.28:0/1827866519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e40a5dd0 con 0x7f36f41442b0 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- 172.21.15.28:0/1827866519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 msgr2=0x7f36f40b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 --2- 172.21.15.28:0/1827866519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f40b3dc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009d90 tx=0x7f36e401c940 comp rx=0 tx=0).stop 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- 172.21.15.28:0/1827866519 shutdown_connections 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 --2- 172.21.15.28:0/1827866519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f40b3dc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- 172.21.15.28:0/1827866519 >> 172.21.15.28:0/1827866519 conn(0x7f36f40674f0 msgr2=0x7f36f406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- 172.21.15.28:0/1827866519 shutdown_connections 2024-03-21T08:41:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- 172.21.15.28:0/1827866519 wait complete. 2024-03-21T08:41:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 Processor -- start 2024-03-21T08:41:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- start start 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f36f4071a80 0x7f36f41fa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36f4144690 0x7f36f413d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f406e430 con 0x7f36f41442b0 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f406e2b0 con 0x7f36f4144690 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36fa200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f406e5b0 con 0x7f36f4071a80 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55694/0 (socket says 172.21.15.28:55694) 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36eaa00640 1 -- 172.21.15.28:0/769134054 learned_addr learned my addr 172.21.15.28:0/769134054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.201+0000 7f36f3e00640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36f4144690 0x7f36f413d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36f3400640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f36f4071a80 0x7f36f41fa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 -- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f36f4071a80 msgr2=0x7f36f41fa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f36f4071a80 0x7f36f41fa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 -- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36f4144690 msgr2=0x7f36f413d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36f4144690 0x7f36f413d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 -- 172.21.15.28:0/769134054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f4142600 con 0x7f36f41442b0 2024-03-21T08:41:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36eaa00640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00c440 tx=0x7f36dc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc018070 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0074e0 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc004a10 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f41fae20 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f40af9c0 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f4073430 con 0x7f36f41442b0 2024-03-21T08:41:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.205+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f36dc004bb0 con 0x7f36f41442b0 2024-03-21T08:41:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.209+0000 7f36ebe00640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 0x7f36d006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.209+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f36dc085350 con 0x7f36f41442b0 2024-03-21T08:41:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.209+0000 7f36f3400640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 0x7f36d006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.209+0000 7f36f3400640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 0x7f36d006a010 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36e40af000 tx=0x7f36e40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.213+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f36dc0553c0 con 0x7f36f41442b0 2024-03-21T08:41:33.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36f4133e90 con 0x7f36f41442b0 2024-03-21T08:41:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36ebe00640 1 -- 172.21.15.28:0/769134054 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36dc059460 con 0x7f36f41442b0 2024-03-21T08:41:33.382 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 msgr2=0x7f36d006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 0x7f36d006a010 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36e40af000 tx=0x7f36e40a7070 comp rx=0 tx=0).stop 2024-03-21T08:41:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 msgr2=0x7f36f41fa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00c440 tx=0x7f36dc00c900 comp rx=0 tx=0).stop 2024-03-21T08:41:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 shutdown_connections 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f36f4144690 0x7f36f413d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f41442b0 0x7f36f41fa810 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f36d0067b50 0x7f36d006a010 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 --2- 172.21.15.28:0/769134054 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f36f4071a80 0x7f36f41fa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 >> 172.21.15.28:0/769134054 conn(0x7f36f40674f0 msgr2=0x7f36f41690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 shutdown_connections 2024-03-21T08:41:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.377+0000 7f36fa200640 1 -- 172.21.15.28:0/769134054 wait complete. 2024-03-21T08:41:33.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 Processor -- start 2024-03-21T08:41:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 -- start start 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe5c8071a80 0x7fe5c806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 0x7fe5c80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c80ce5b0 con 0x7fe5c80cde40 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c815c720 con 0x7fe5c8071a80 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5ce400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80ce890 con 0x7fe5c815cbb0 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5bfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe5c80cde40 0x7fe5c8071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38656/0 (socket says 172.21.15.28:38656) 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5bfe00640 1 -- 172.21.15.28:0/477743849 learned_addr learned my addr 172.21.15.28:0/477743849 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7e00640 1 --2- 172.21.15.28:0/477743849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 0x7fe5c80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7400640 1 --2- 172.21.15.28:0/477743849 >> v2:172.21.15.151:3300/0 conn(0x7fe5c8071a80 0x7fe5c806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c6a00640 1 -- 172.21.15.28:0/477743849 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227232644 0 0) 0x7fe5c80ce5b0 con 0x7fe5c80cde40 2024-03-21T08:41:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7e00640 1 -- 172.21.15.28:0/477743849 >> v2:172.21.15.151:3300/0 conn(0x7fe5c8071a80 msgr2=0x7fe5c806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7e00640 1 --2- 172.21.15.28:0/477743849 >> v2:172.21.15.151:3300/0 conn(0x7fe5c8071a80 0x7fe5c806f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7e00640 1 -- 172.21.15.28:0/477743849 >> v1:172.21.15.28:6789/0 conn(0x7fe5c80cde40 legacy=0x7fe5c8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.541+0000 7fe5c7e00640 1 -- 172.21.15.28:0/477743849 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c815c3b0 con 0x7fe5c815cbb0 2024-03-21T08:41:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5c7e00640 1 --2- 172.21.15.28:0/477743849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 0x7fe5c80cc4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009d90 tx=0x7fe5c001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151e70ff2e9fbfac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5c6a00640 1 -- 172.21.15.28:0/477743849 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00b2070 con 0x7fe5c815cbb0 2024-03-21T08:41:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5c6a00640 1 -- 172.21.15.28:0/477743849 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c00a5980 con 0x7fe5c815cbb0 2024-03-21T08:41:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5c6a00640 1 -- 172.21.15.28:0/477743849 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00a5d50 con 0x7fe5c815cbb0 2024-03-21T08:41:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 -- 172.21.15.28:0/477743849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 msgr2=0x7fe5c80cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 --2- 172.21.15.28:0/477743849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 0x7fe5c80cc4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009d90 tx=0x7fe5c001c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 -- 172.21.15.28:0/477743849 shutdown_connections 2024-03-21T08:41:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 --2- 172.21.15.28:0/477743849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c815cbb0 0x7fe5c80cc4b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 --2- 172.21.15.28:0/477743849 >> v2:172.21.15.151:3300/0 conn(0x7fe5c8071a80 0x7fe5c806f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 -- 172.21.15.28:0/477743849 >> 172.21.15.28:0/477743849 conn(0x7fe5c80674f0 msgr2=0x7fe5c806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 -- 172.21.15.28:0/477743849 shutdown_connections 2024-03-21T08:41:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.545+0000 7fe5ce400640 1 -- 172.21.15.28:0/477743849 wait complete. 2024-03-21T08:41:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 Processor -- start 2024-03-21T08:41:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- start start 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 0x7fe5c813cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5c815cbb0 0x7fe5c813a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8168f20 con 0x7fe5c80cde40 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8144b20 con 0x7fe5c815cbb0 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c81690a0 con 0x7fe5c8071a80 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55696/0 (socket says 172.21.15.28:55696) 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3090507422 learned_addr learned my addr 172.21.15.28:0/3090507422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5c7400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 0x7fe5c813cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5c7e00640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5c815cbb0 0x7fe5c813a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 msgr2=0x7fe5c813cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 0x7fe5c813cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5c815cbb0 msgr2=0x7fe5c813a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5c815cbb0 0x7fe5c813a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3090507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80ae680 con 0x7fe5c80cde40 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5c7400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 0x7fe5c813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bfe00640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00011f0 tx=0x7fe5b000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0018070 con 0x7fe5c80cde40 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b00074e0 con 0x7fe5c80cde40 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c80b2fa0 con 0x7fe5c80cde40 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0004a10 con 0x7fe5c80cde40 2024-03-21T08:41:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.549+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c80b3560 con 0x7fe5c80cde40 2024-03-21T08:41:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c8073430 con 0x7fe5c80cde40 2024-03-21T08:41:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe5b001b070 con 0x7fe5c80cde40 2024-03-21T08:41:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5bf400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 0x7fe5a4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe5b00854a0 con 0x7fe5c80cde40 2024-03-21T08:41:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5c7400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 0x7fe5a4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.553+0000 7fe5c7400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 0x7fe5a4069fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5c813e490 tx=0x7fe5b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.557+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe5b0055510 con 0x7fe5c80cde40 2024-03-21T08:41:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.721+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5c813e560 con 0x7fe5c80cde40 2024-03-21T08:41:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.721+0000 7fe5bf400640 1 -- 172.21.15.28:0/3090507422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe5b00595b0 con 0x7fe5c80cde40 2024-03-21T08:41:33.726 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 msgr2=0x7fe5a4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 0x7fe5a4069fe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5c813e490 tx=0x7fe5b8009190 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 msgr2=0x7fe5c813a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00011f0 tx=0x7fe5b000c9b0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 shutdown_connections 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe5c815cbb0 0x7fe5c813a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c80cde40 0x7fe5c813a120 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe5a4067b20 0x7fe5a4069fe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 --2- 172.21.15.28:0/3090507422 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe5c8071a80 0x7fe5c813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 >> 172.21.15.28:0/3090507422 conn(0x7fe5c80674f0 msgr2=0x7fe5c813ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 shutdown_connections 2024-03-21T08:41:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.725+0000 7fe5ce400640 1 -- 172.21.15.28:0/3090507422 wait complete. 2024-03-21T08:41:33.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:33.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.885+0000 7f50fd800640 1 Processor -- start 2024-03-21T08:41:33.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 -- start start 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 0x7f50f80cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f50f815cbb0 0x7f50f80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f806f500 con 0x7f50f8071a80 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f80ce380 con 0x7f50f815cbb0 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50fd800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f815c720 con 0x7f50f806f0a0 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50efe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 0x7f50f80cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f50f815cbb0 0x7f50f80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f50f815cbb0 0x7f50f80cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:51088/0 (socket says 172.21.15.28:51088) 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 -- 172.21.15.28:0/3940915595 learned_addr learned my addr 172.21.15.28:0/3940915595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50efe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 0x7f50f80cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48616/0 (socket says 172.21.15.28:48616) 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 -- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 msgr2=0x7f50f80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 --2- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 0x7f50f80cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 -- 172.21.15.28:0/3940915595 >> v1:172.21.15.151:6789/0 conn(0x7f50f8071a80 legacy=0x7f50f8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.889+0000 7f50f7400640 1 -- 172.21.15.28:0/3940915595 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80ce5b0 con 0x7f50f815cbb0 2024-03-21T08:41:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50f7400640 1 --2- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f815cbb0 0x7f50f80cc4b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009e50 tx=0x7f50f001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6697cb16d4cee97b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50f6000640 1 -- 172.21.15.28:0/3940915595 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00a5830 con 0x7f50f815cbb0 2024-03-21T08:41:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50f6000640 1 -- 172.21.15.28:0/3940915595 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00a59d0 con 0x7f50f815cbb0 2024-03-21T08:41:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50f6000640 1 -- 172.21.15.28:0/3940915595 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00a5da0 con 0x7f50f815cbb0 2024-03-21T08:41:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f815cbb0 msgr2=0x7f50f80cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 --2- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f815cbb0 0x7f50f80cc4b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009e50 tx=0x7f50f001c970 comp rx=0 tx=0).stop 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- 172.21.15.28:0/3940915595 shutdown_connections 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 --2- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f815cbb0 0x7f50f80cc4b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 --2- 172.21.15.28:0/3940915595 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f806f0a0 0x7f50f80cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- 172.21.15.28:0/3940915595 >> 172.21.15.28:0/3940915595 conn(0x7f50f80674f0 msgr2=0x7f50f806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- 172.21.15.28:0/3940915595 shutdown_connections 2024-03-21T08:41:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- 172.21.15.28:0/3940915595 wait complete. 2024-03-21T08:41:33.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 Processor -- start 2024-03-21T08:41:33.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.893+0000 7f50fd800640 1 -- start start 2024-03-21T08:41:33.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f806f0a0 0x7f50f8142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f8071a80 0x7f50f81584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8144b20 con 0x7f50f815cbb0 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80cce00 con 0x7f50f806f0a0 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8168f20 con 0x7f50f8071a80 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55712/0 (socket says 172.21.15.28:55712) 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 -- 172.21.15.28:0/2914458833 learned_addr learned my addr 172.21.15.28:0/2914458833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50efe00640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f8071a80 0x7f50f81584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f6a00640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f806f0a0 0x7f50f8142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 -- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f8071a80 msgr2=0x7f50f81584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f8071a80 0x7f50f81584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 -- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f806f0a0 msgr2=0x7f50f8142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f806f0a0 0x7f50f8142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 -- 172.21.15.28:0/2914458833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80ad900 con 0x7f50f815cbb0 2024-03-21T08:41:33.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50f7400640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009e50 tx=0x7f50f0019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00bc020 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00b8670 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f80adbe0 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0009510 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f8138d70 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f8073430 con 0x7f50f815cbb0 2024-03-21T08:41:33.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.897+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f50f00bf020 con 0x7f50f815cbb0 2024-03-21T08:41:33.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.901+0000 7f50eea00640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 0x7f50d4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:33.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.901+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f50f0121180 con 0x7f50f815cbb0 2024-03-21T08:41:33.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.901+0000 7f50f6a00640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 0x7f50d4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:33.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.901+0000 7f50f6a00640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 0x7f50d4069fe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50e8001810 tx=0x7f50e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:33.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:33.905+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f50f00c6030 con 0x7f50f815cbb0 2024-03-21T08:41:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f50f804f4f0 con 0x7f50f815cbb0 2024-03-21T08:41:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50eea00640 1 -- 172.21.15.28:0/2914458833 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f50f0016620 con 0x7f50f815cbb0 2024-03-21T08:41:34.074 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 msgr2=0x7f50d4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 0x7f50d4069fe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50e8001810 tx=0x7f50e8009040 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 msgr2=0x7f50f81589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009e50 tx=0x7f50f0019710 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 shutdown_connections 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50f815cbb0 0x7f50f81589e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f50d4067b20 0x7f50d4069fe0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f50f8071a80 0x7f50f81584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 --2- 172.21.15.28:0/2914458833 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f50f806f0a0 0x7f50f8142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.069+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 >> 172.21.15.28:0/2914458833 conn(0x7f50f80674f0 msgr2=0x7f50f80b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.073+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 shutdown_connections 2024-03-21T08:41:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.073+0000 7f50fd800640 1 -- 172.21.15.28:0/2914458833 wait complete. 2024-03-21T08:41:34.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.233+0000 7faddce00640 1 Processor -- start 2024-03-21T08:41:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.233+0000 7faddce00640 1 -- start start 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7faddce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadd8071a80 0x7fadd806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7faddce00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 0x7fadd8071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7faddce00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd80723a0 con 0x7fadd815cbb0 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7faddce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd815c720 con 0x7fadd8071a80 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7faddce00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ce5b0 con 0x7fadd80cde40 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadd8071a80 0x7fadd806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadd8071a80 0x7fadd806f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55728/0 (socket says 172.21.15.28:55728) 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 -- 172.21.15.28:0/3069334306 learned_addr learned my addr 172.21.15.28:0/3069334306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadcd400640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 0x7fadd8071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 -- 172.21.15.28:0/3069334306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 msgr2=0x7fadd8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 0x7fadd8071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 -- 172.21.15.28:0/3069334306 >> v1:172.21.15.151:6789/0 conn(0x7fadd815cbb0 legacy=0x7fadd80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadd5e00640 1 -- 172.21.15.28:0/3069334306 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80ce890 con 0x7fadd8071a80 2024-03-21T08:41:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.237+0000 7fadcd400640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 0x7fadd8071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7fadd5e00640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd806f0a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009e50 tx=0x7fadc801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0fe40dfc4d7456 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7fadd5400640 1 -- 172.21.15.28:0/3069334306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80a5850 con 0x7fadd8071a80 2024-03-21T08:41:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7fadd5400640 1 -- 172.21.15.28:0/3069334306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc80a59f0 con 0x7fadd8071a80 2024-03-21T08:41:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7fadd5400640 1 -- 172.21.15.28:0/3069334306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80a5dc0 con 0x7fadd8071a80 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- 172.21.15.28:0/3069334306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 msgr2=0x7fadd806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd806f0a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009e50 tx=0x7fadc801c970 comp rx=0 tx=0).stop 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- 172.21.15.28:0/3069334306 shutdown_connections 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd80cde40 0x7fadd8071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- 172.21.15.28:0/3069334306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd806f0a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- 172.21.15.28:0/3069334306 >> 172.21.15.28:0/3069334306 conn(0x7fadd80674f0 msgr2=0x7fadd806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- 172.21.15.28:0/3069334306 shutdown_connections 2024-03-21T08:41:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- 172.21.15.28:0/3069334306 wait complete. 2024-03-21T08:41:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 Processor -- start 2024-03-21T08:41:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- start start 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 0x7fadd81f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 0x7fadd81f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8168f20 con 0x7fadd8071a80 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ce220 con 0x7fadd80cde40 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7faddce00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81690a0 con 0x7fadd815cbb0 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.241+0000 7fadd5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55730/0 (socket says 172.21.15.28:55730) 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 -- 172.21.15.28:0/3502383043 learned_addr learned my addr 172.21.15.28:0/3502383043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd6800640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 0x7fadd81f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcd400640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 0x7fadd81f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 -- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 msgr2=0x7fadd81f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 0x7fadd81f8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 -- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 msgr2=0x7fadd81f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 0x7fadd81f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 -- 172.21.15.28:0/3502383043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80ace20 con 0x7fadd8071a80 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd6800640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 0x7fadd81f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcd400640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 0x7fadd81f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadd5e00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadc801a5a0 tx=0x7fadc8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80bc020 con 0x7fadd8071a80 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc80b8670 con 0x7fadd8071a80 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd80ad100 con 0x7fadd8071a80 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80b8a20 con 0x7fadd8071a80 2024-03-21T08:41:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd80ad6c0 con 0x7fadd8071a80 2024-03-21T08:41:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd8073430 con 0x7fadd8071a80 2024-03-21T08:41:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fadc80bf020 con 0x7fadd8071a80 2024-03-21T08:41:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 0x7fadb4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.245+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fadc81214a0 con 0x7fadd8071a80 2024-03-21T08:41:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.249+0000 7fadcd400640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 0x7fadb4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.249+0000 7fadcd400640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 0x7fadb4069fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fadd81f8510 tx=0x7fadc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.249+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fadc80c6030 con 0x7fadd8071a80 2024-03-21T08:41:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.413+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fadd804f4f0 con 0x7fadd8071a80 2024-03-21T08:41:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.413+0000 7fadcea00640 1 -- 172.21.15.28:0/3502383043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fadc8016620 con 0x7fadd8071a80 2024-03-21T08:41:34.419 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 msgr2=0x7fadb4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 0x7fadb4069fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fadd81f8510 tx=0x7fadc0008040 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 msgr2=0x7fadd81f7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadc801a5a0 tx=0x7fadc8019710 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 shutdown_connections 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fadb4067b20 0x7fadb4069fe0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fadd815cbb0 0x7fadd81f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fadd80cde40 0x7fadd81f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 --2- 172.21.15.28:0/3502383043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd8071a80 0x7fadd81f7bd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 >> 172.21.15.28:0/3502383043 conn(0x7fadd80674f0 msgr2=0x7fadd80cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 shutdown_connections 2024-03-21T08:41:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.417+0000 7faddce00640 1 -- 172.21.15.28:0/3502383043 wait complete. 2024-03-21T08:41:34.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 Processor -- start 2024-03-21T08:41:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 -- start start 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cedb0 0x7ff2100cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2100cf970 0x7ff2100cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff210160590 con 0x7ff21015bbe0 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21015fd20 con 0x7ff2100cf970 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff215c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21015ffd0 con 0x7ff2100cedb0 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2100cf970 0x7ff2100cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2100cf970 0x7ff2100cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55740/0 (socket says 172.21.15.28:55740) 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 -- 172.21.15.28:0/3205233847 learned_addr learned my addr 172.21.15.28:0/3205233847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20ea00640 1 --1- 172.21.15.28:0/3205233847 >> v1:172.21.15.28:6789/0 conn(0x7ff21015bbe0 0x7ff21015f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38660/0 (socket says 172.21.15.28:38660) 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff207e00640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cedb0 0x7ff2100cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 -- 172.21.15.28:0/3205233847 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cedb0 msgr2=0x7ff2100cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cedb0 0x7ff2100cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 -- 172.21.15.28:0/3205233847 >> v1:172.21.15.28:6789/0 conn(0x7ff21015bbe0 legacy=0x7ff21015f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20f400640 1 -- 172.21.15.28:0/3205233847 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2101602b0 con 0x7ff2100cf970 2024-03-21T08:41:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.577+0000 7ff20e000640 1 -- 172.21.15.28:0/3205233847 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922515534 0 0) 0x7ff210160590 con 0x7ff21015bbe0 2024-03-21T08:41:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff20f400640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2100cf970 0x7ff2100cfe00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac6ca89f71b8702 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff20e000640 1 -- 172.21.15.28:0/3205233847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080a5830 con 0x7ff2100cf970 2024-03-21T08:41:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff20e000640 1 -- 172.21.15.28:0/3205233847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2080a59d0 con 0x7ff2100cf970 2024-03-21T08:41:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff20e000640 1 -- 172.21.15.28:0/3205233847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080a5da0 con 0x7ff2100cf970 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 -- 172.21.15.28:0/3205233847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2100cf970 msgr2=0x7ff2100cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2100cf970 0x7ff2100cfe00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff208009e50 tx=0x7ff20801c970 comp rx=0 tx=0).stop 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 -- 172.21.15.28:0/3205233847 shutdown_connections 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2100cf970 0x7ff2100cfe00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 --2- 172.21.15.28:0/3205233847 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cedb0 0x7ff2100cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 -- 172.21.15.28:0/3205233847 >> 172.21.15.28:0/3205233847 conn(0x7ff210099a30 msgr2=0x7ff210099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 -- 172.21.15.28:0/3205233847 shutdown_connections 2024-03-21T08:41:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.581+0000 7ff215c00640 1 -- 172.21.15.28:0/3205233847 wait complete. 2024-03-21T08:41:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 Processor -- start 2024-03-21T08:41:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- start start 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2100cedb0 0x7ff210155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 0x7ff2100b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101689b0 con 0x7ff21015bbe0 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100ce060 con 0x7ff2100cedb0 2024-03-21T08:41:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff210168b30 con 0x7ff2100cf970 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff207e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 0x7ff2100b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2100cedb0 0x7ff210155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff207e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 0x7ff2100b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:48668/0 (socket says 172.21.15.28:48668) 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff207e00640 1 -- 172.21.15.28:0/2875010320 learned_addr learned my addr 172.21.15.28:0/2875010320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55756/0 (socket says 172.21.15.28:55756) 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 -- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 msgr2=0x7ff2100b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 0x7ff2100b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 -- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2100cedb0 msgr2=0x7ff210155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2100cedb0 0x7ff210155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 -- 172.21.15.28:0/2875010320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100b0d60 con 0x7ff21015bbe0 2024-03-21T08:41:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff20f400640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2080ab000 tx=0x7ff208019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080bc030 con 0x7ff21015bbe0 2024-03-21T08:41:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2080b8470 con 0x7ff21015bbe0 2024-03-21T08:41:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080b8860 con 0x7ff21015bbe0 2024-03-21T08:41:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100b1040 con 0x7ff21015bbe0 2024-03-21T08:41:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2101562f0 con 0x7ff21015bbe0 2024-03-21T08:41:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.585+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2100b1560 con 0x7ff21015bbe0 2024-03-21T08:41:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.589+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff2080bf070 con 0x7ff21015bbe0 2024-03-21T08:41:34.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.589+0000 7ff206a00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 0x7ff1ec06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.589+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff208120c20 con 0x7ff21015bbe0 2024-03-21T08:41:34.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.589+0000 7ff20ea00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 0x7ff1ec06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.589+0000 7ff20ea00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 0x7ff1ec06a010 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f70 tx=0x7ff200007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.593+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff208016620 con 0x7ff21015bbe0 2024-03-21T08:41:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.789+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff21004f410 con 0x7ff21015bbe0 2024-03-21T08:41:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff206a00640 1 -- 172.21.15.28:0/2875010320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7ff2080f0d20 con 0x7ff21015bbe0 2024-03-21T08:41:34.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:34.796 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 msgr2=0x7ff1ec06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 0x7ff1ec06a010 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff200000f70 tx=0x7ff200007df0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 msgr2=0x7ff2101f9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2080ab000 tx=0x7ff208019710 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 shutdown_connections 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff21015bbe0 0x7ff2101f9cd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1ec067b50 0x7ff1ec06a010 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff2100cf970 0x7ff2100b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 --2- 172.21.15.28:0/2875010320 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2100cedb0 0x7ff210155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 >> 172.21.15.28:0/2875010320 conn(0x7ff210099a30 msgr2=0x7ff2100af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 shutdown_connections 2024-03-21T08:41:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.793+0000 7ff215c00640 1 -- 172.21.15.28:0/2875010320 wait complete. 2024-03-21T08:41:34.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:34.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 Processor -- start 2024-03-21T08:41:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 -- start start 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1a140ce490 0x7f1a1415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a1415ba70 0x7f1a140ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a14071980 0x7f1a14071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a1415bf00 con 0x7f1a14071980 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a140cedb0 con 0x7f1a140ce490 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a18e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1415ebf0 con 0x7f1a1415ba70 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a14071980 0x7f1a14071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a14071980 0x7f1a14071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55768/0 (socket says 172.21.15.28:55768) 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 -- 172.21.15.28:0/3176938947 learned_addr learned my addr 172.21.15.28:0/3176938947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a11e00640 1 --2- 172.21.15.28:0/3176938947 >> v2:172.21.15.151:3300/0 conn(0x7f1a140ce490 0x7f1a1415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a09400640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a1415ba70 0x7f1a140ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 -- 172.21.15.28:0/3176938947 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a1415ba70 msgr2=0x7f1a140ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a1415ba70 0x7f1a140ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 -- 172.21.15.28:0/3176938947 >> v2:172.21.15.151:3300/0 conn(0x7f1a140ce490 msgr2=0x7f1a1415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 --2- 172.21.15.28:0/3176938947 >> v2:172.21.15.151:3300/0 conn(0x7f1a140ce490 0x7f1a1415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.953+0000 7f1a12800640 1 -- 172.21.15.28:0/3176938947 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1415eed0 con 0x7f1a14071980 2024-03-21T08:41:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a12800640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a14071980 0x7f1a14071df0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a140730c0 tx=0x7f1a0c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45f8c9b6894ab99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a11400640 1 -- 172.21.15.28:0/3176938947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0b0070 con 0x7f1a14071980 2024-03-21T08:41:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a11400640 1 -- 172.21.15.28:0/3176938947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0c0ab600 con 0x7f1a14071980 2024-03-21T08:41:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a11400640 1 -- 172.21.15.28:0/3176938947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0ab9d0 con 0x7f1a14071980 2024-03-21T08:41:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 -- 172.21.15.28:0/3176938947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a14071980 msgr2=0x7f1a14071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a14071980 0x7f1a14071df0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a140730c0 tx=0x7f1a0c01c490 comp rx=0 tx=0).stop 2024-03-21T08:41:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 -- 172.21.15.28:0/3176938947 shutdown_connections 2024-03-21T08:41:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a14071980 0x7f1a14071df0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 --2- 172.21.15.28:0/3176938947 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a1415ba70 0x7f1a140ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 --2- 172.21.15.28:0/3176938947 >> v2:172.21.15.151:3300/0 conn(0x7f1a140ce490 0x7f1a1415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 -- 172.21.15.28:0/3176938947 >> 172.21.15.28:0/3176938947 conn(0x7f1a140cc230 msgr2=0x7f1a140cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 -- 172.21.15.28:0/3176938947 shutdown_connections 2024-03-21T08:41:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.957+0000 7f1a18e00640 1 -- 172.21.15.28:0/3176938947 wait complete. 2024-03-21T08:41:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 Processor -- start 2024-03-21T08:41:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- start start 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a14071980 0x7f1a14144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a1415ba70 0x7f1a1406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14155390 con 0x7f1a140ce490 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14072740 con 0x7f1a1415ba70 2024-03-21T08:41:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140b4b90 con 0x7f1a14071980 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55782/0 (socket says 172.21.15.28:55782) 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 -- 172.21.15.28:0/491360003 learned_addr learned my addr 172.21.15.28:0/491360003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a11e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a14071980 0x7f1a14144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a12800640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a1415ba70 0x7f1a1406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 -- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a14071980 msgr2=0x7f1a14144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a14071980 0x7f1a14144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 -- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a1415ba70 msgr2=0x7f1a1406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a1415ba70 0x7f1a1406ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 -- 172.21.15.28:0/491360003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140ab7d0 con 0x7f1a140ce490 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a09400640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00e8a0 tx=0x7f19fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc017070 con 0x7f1a140ce490 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc00c940 con 0x7f1a140ce490 2024-03-21T08:41:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a140a9810 con 0x7f1a140ce490 2024-03-21T08:41:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc00ccf0 con 0x7f1a140ce490 2024-03-21T08:41:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.961+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a140a9dd0 con 0x7f1a140ce490 2024-03-21T08:41:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1404f410 con 0x7f1a140ce490 2024-03-21T08:41:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f19fc01a070 con 0x7f1a140ce490 2024-03-21T08:41:34.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a0aa00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 0x7f19f0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:34.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f19fc086180 con 0x7f1a140ce490 2024-03-21T08:41:34.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a11e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 0x7f19f0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:34.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.965+0000 7f1a11e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 0x7f19f0069fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a141fa410 tx=0x7f1a0400d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:34.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:34.969+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1a140ab7d0 con 0x7f1a140ce490 2024-03-21T08:41:35.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.133+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a14072c10 con 0x7f1a140ce490 2024-03-21T08:41:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.133+0000 7f1a0aa00640 1 -- 172.21.15.28:0/491360003 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7f19fc056160 con 0x7f1a140ce490 2024-03-21T08:41:35.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:35.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 msgr2=0x7f19f0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 0x7f19f0069fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a141fa410 tx=0x7f1a0400d040 comp rx=0 tx=0).stop 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 msgr2=0x7f1a140a8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00e8a0 tx=0x7f19fc00eca0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 shutdown_connections 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1a1415ba70 0x7f1a1406ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a140ce490 0x7f1a140a8ea0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19f0067b20 0x7f19f0069fe0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 --2- 172.21.15.28:0/491360003 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1a14071980 0x7f1a14144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 >> 172.21.15.28:0/491360003 conn(0x7f1a140cc230 msgr2=0x7f1a1406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 shutdown_connections 2024-03-21T08:41:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.137+0000 7f1a18e00640 1 -- 172.21.15.28:0/491360003 wait complete. 2024-03-21T08:41:35.162 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:35.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 Processor -- start 2024-03-21T08:41:35.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 -- start start 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5e58072190 0x7f5e580ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e580cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e580cd600 con 0x7f5e580b43b0 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e580b3ae0 con 0x7f5e58072190 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e5ca00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580cd870 con 0x7f5e580b4790 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e55e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5e580b43b0 0x7f5e58071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38668/0 (socket says 172.21.15.28:38668) 2024-03-21T08:41:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e55e00640 1 -- 172.21.15.28:0/967001770 learned_addr learned my addr 172.21.15.28:0/967001770 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967001770 >> v2:172.21.15.151:3300/0 conn(0x7f5e58072190 0x7f5e580ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e56800640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e580cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e55400640 1 -- 172.21.15.28:0/967001770 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909949545 0 0) 0x7f5e580cd600 con 0x7f5e580b43b0 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e55400640 1 -- 172.21.15.28:0/967001770 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c005740 con 0x7f5e580b43b0 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e4d400640 1 -- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 msgr2=0x7f5e580cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e580cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e4d400640 1 -- 172.21.15.28:0/967001770 >> v1:172.21.15.28:6789/0 conn(0x7f5e580b43b0 legacy=0x7f5e58071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.305+0000 7f5e4d400640 1 -- 172.21.15.28:0/967001770 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5815b190 con 0x7f5e58072190 2024-03-21T08:41:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e58072190 0x7f5e580ccb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e4001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1965ea067e375515 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e55400640 1 -- 172.21.15.28:0/967001770 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400a7070 con 0x7f5e58072190 2024-03-21T08:41:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e55400640 1 -- 172.21.15.28:0/967001770 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e400b0070 con 0x7f5e58072190 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e55400640 1 -- 172.21.15.28:0/967001770 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400ab890 con 0x7f5e58072190 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e58072190 msgr2=0x7f5e580ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e58072190 0x7f5e580ccb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e4001c470 comp rx=0 tx=0).stop 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967001770 shutdown_connections 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e580cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967001770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e58072190 0x7f5e580ccb80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967001770 >> 172.21.15.28:0/967001770 conn(0x7f5e580674f0 msgr2=0x7f5e5806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967001770 shutdown_connections 2024-03-21T08:41:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.309+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967001770 wait complete. 2024-03-21T08:41:35.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 Processor -- start 2024-03-21T08:41:35.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- start start 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e580b43b0 0x7f5e58164d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e5814e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580b32e0 con 0x7f5e58072190 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5815bf50 con 0x7f5e580b43b0 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580b3460 con 0x7f5e580b4790 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55790/0 (socket says 172.21.15.28:55790) 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 -- 172.21.15.28:0/967195469 learned_addr learned my addr 172.21.15.28:0/967195469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e580b43b0 0x7f5e58164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e56800640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e5814e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 -- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 msgr2=0x7f5e5814e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e5814e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 -- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e580b43b0 msgr2=0x7f5e58164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e580b43b0 0x7f5e58164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 -- 172.21.15.28:0/967195469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e58152240 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e55e00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800cb70 tx=0x7f5e4800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48017070 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4800ad40 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48012620 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e581408b0 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.313+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e58152a70 con 0x7f5e58072190 2024-03-21T08:41:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e58073430 con 0x7f5e58072190 2024-03-21T08:41:35.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5e4801a070 con 0x7f5e58072190 2024-03-21T08:41:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e4ea00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 0x7f5e3406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5e48085f00 con 0x7f5e58072190 2024-03-21T08:41:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 0x7f5e3406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.317+0000 7f5e4d400640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 0x7f5e3406a010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e400af000 tx=0x7f5e400a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.321+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5e48056010 con 0x7f5e58072190 2024-03-21T08:41:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e5804f4f0 con 0x7f5e58072190 2024-03-21T08:41:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e4ea00640 1 -- 172.21.15.28:0/967195469 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e4805a0b0 con 0x7f5e58072190 2024-03-21T08:41:35.492 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 msgr2=0x7f5e3406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 0x7f5e3406a010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e400af000 tx=0x7f5e400a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 msgr2=0x7f5e58140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800cb70 tx=0x7f5e4800a430 comp rx=0 tx=0).stop 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 shutdown_connections 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e34067b50 0x7f5e3406a010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e580b4790 0x7f5e5814e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e580b43b0 0x7f5e58164d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 --2- 172.21.15.28:0/967195469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58072190 0x7f5e58140370 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 >> 172.21.15.28:0/967195469 conn(0x7f5e580674f0 msgr2=0x7f5e5814a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 shutdown_connections 2024-03-21T08:41:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.489+0000 7f5e5ca00640 1 -- 172.21.15.28:0/967195469 wait complete. 2024-03-21T08:41:35.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:35.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 Processor -- start 2024-03-21T08:41:35.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 -- start start 2024-03-21T08:41:35.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2200b4420 0x7fd220071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22015bf90 con 0x7fd2200cd300 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22015b1d0 con 0x7fd2200b4800 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.653+0000 7fd226e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22015b4b0 con 0x7fd2200b4420 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2200b4420 0x7fd220071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd224e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd2200cd300 0x7fd2200b2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38678/0 (socket says 172.21.15.28:38678) 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd217e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd2200b4800 0x7fd2200ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:51228/0 (socket says 172.21.15.28:51228) 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2200b4420 0x7fd220071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55802/0 (socket says 172.21.15.28:55802) 2024-03-21T08:41:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd224e00640 1 -- 172.21.15.28:0/2168361846 learned_addr learned my addr 172.21.15.28:0/2168361846 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:35.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 -- 172.21.15.28:0/2168361846 >> v1:172.21.15.151:6789/0 conn(0x7fd2200b4800 legacy=0x7fd2200ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 -- 172.21.15.28:0/2168361846 >> v1:172.21.15.28:6789/0 conn(0x7fd2200cd300 legacy=0x7fd2200b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 -- 172.21.15.28:0/2168361846 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22015b790 con 0x7fd2200b4420 2024-03-21T08:41:35.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21f400640 1 -- 172.21.15.28:0/2168361846 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717625276 0 0) 0x7fd22015bf90 con 0x7fd2200cd300 2024-03-21T08:41:35.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21fe00640 1 --2- 172.21.15.28:0/2168361846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2200b4420 0x7fd220071a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2100097c0 tx=0x7fd21001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2650d552720563 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21f400640 1 -- 172.21.15.28:0/2168361846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100b2070 con 0x7fd2200b4420 2024-03-21T08:41:35.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21f400640 1 -- 172.21.15.28:0/2168361846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2100ad680 con 0x7fd2200b4420 2024-03-21T08:41:35.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.657+0000 7fd21f400640 1 -- 172.21.15.28:0/2168361846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100ada50 con 0x7fd2200b4420 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- 172.21.15.28:0/2168361846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2200b4420 msgr2=0x7fd220071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 --2- 172.21.15.28:0/2168361846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2200b4420 0x7fd220071a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2100097c0 tx=0x7fd21001c470 comp rx=0 tx=0).stop 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- 172.21.15.28:0/2168361846 shutdown_connections 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 --2- 172.21.15.28:0/2168361846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2200b4420 0x7fd220071a80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- 172.21.15.28:0/2168361846 >> 172.21.15.28:0/2168361846 conn(0x7fd2200674f0 msgr2=0x7fd22006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- 172.21.15.28:0/2168361846 shutdown_connections 2024-03-21T08:41:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- 172.21.15.28:0/2168361846 wait complete. 2024-03-21T08:41:35.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 Processor -- start 2024-03-21T08:41:35.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- start start 2024-03-21T08:41:35.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2200b4420 0x7fd22013feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2200cd300 0x7fd2201429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd220168b50 con 0x7fd220142ef0 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2201689d0 con 0x7fd2200b4420 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd226e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd220168cd0 con 0x7fd2200cd300 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55808/0 (socket says 172.21.15.28:55808) 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 -- 172.21.15.28:0/3658823606 learned_addr learned my addr 172.21.15.28:0/3658823606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd21fe00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2200b4420 0x7fd22013feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd217e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2200cd300 0x7fd2201429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 -- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2200cd300 msgr2=0x7fd2201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2200cd300 0x7fd2201429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 -- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2200b4420 msgr2=0x7fd22013feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2200b4420 0x7fd22013feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 -- 172.21.15.28:0/3658823606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22014f520 con 0x7fd220142ef0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.661+0000 7fd224e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c780 tx=0x7fd21800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218018070 con 0x7fd220142ef0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218004d70 con 0x7fd220142ef0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21800a880 con 0x7fd220142ef0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22014f800 con 0x7fd220142ef0 2024-03-21T08:41:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22014fdc0 con 0x7fd220142ef0 2024-03-21T08:41:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd220073430 con 0x7fd220142ef0 2024-03-21T08:41:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd21801b070 con 0x7fd220142ef0 2024-03-21T08:41:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 0x7fd1fc06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd218085880 con 0x7fd220142ef0 2024-03-21T08:41:35.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21fe00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 0x7fd1fc06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:35.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.665+0000 7fd21fe00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 0x7fd1fc06a010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2100af000 tx=0x7fd2100a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.669+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd2180557d0 con 0x7fd220142ef0 2024-03-21T08:41:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.833+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2200685b0 con 0x7fd220142ef0 2024-03-21T08:41:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.833+0000 7fd21cc00640 1 -- 172.21.15.28:0/3658823606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd218059870 con 0x7fd220142ef0 2024-03-21T08:41:35.840 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 msgr2=0x7fd1fc06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 0x7fd1fc06a010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2100af000 tx=0x7fd2100a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 msgr2=0x7fd220150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c780 tx=0x7fd21800cc40 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 shutdown_connections 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd220142ef0 0x7fd220150580 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd1fc067b50 0x7fd1fc06a010 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2200cd300 0x7fd2201429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 --2- 172.21.15.28:0/3658823606 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2200b4420 0x7fd22013feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 >> 172.21.15.28:0/3658823606 conn(0x7fd2200674f0 msgr2=0x7fd220154020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 shutdown_connections 2024-03-21T08:41:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.837+0000 7fd226e00640 1 -- 172.21.15.28:0/3658823606 wait complete. 2024-03-21T08:41:35.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.997+0000 7fd9b3000640 1 Processor -- start 2024-03-21T08:41:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:35.997+0000 7fd9b3000640 1 -- start start 2024-03-21T08:41:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b3000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b3000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0cd670 con 0x7fd9ac0b4420 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac15bf90 con 0x7fd9ac0b4800 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b3000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac15b1d0 con 0x7fd9ac072190 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55824/0 (socket says 172.21.15.28:55824) 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 -- 172.21.15.28:0/493394597 learned_addr learned my addr 172.21.15.28:0/493394597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9abe00640 1 --1- 172.21.15.28:0/493394597 >> v1:172.21.15.151:6789/0 conn(0x7fd9ac0b4420 0x7fd9ac071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:51232/0 (socket says 172.21.15.28:51232) 2024-03-21T08:41:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9b1000640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 -- 172.21.15.28:0/493394597 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 msgr2=0x7fd9ac0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac0cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 -- 172.21.15.28:0/493394597 >> v1:172.21.15.151:6789/0 conn(0x7fd9ac0b4420 legacy=0x7fd9ac071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9a3e00640 1 -- 172.21.15.28:0/493394597 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac15b4b0 con 0x7fd9ac0b4800 2024-03-21T08:41:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.001+0000 7fd9ab400640 1 -- 172.21.15.28:0/493394597 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312519516 0 0) 0x7fd9ac0cd670 con 0x7fd9ac0b4420 2024-03-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9a3e00640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4c37a92c875972 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9ab400640 1 -- 172.21.15.28:0/493394597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940ab430 con 0x7fd9ac0b4800 2024-03-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9ab400640 1 -- 172.21.15.28:0/493394597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9940b0070 con 0x7fd9ac0b4800 2024-03-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9ab400640 1 -- 172.21.15.28:0/493394597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940ab7f0 con 0x7fd9ac0b4800 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- 172.21.15.28:0/493394597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac0b4800 msgr2=0x7fd9ac0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99401c530 comp rx=0 tx=0).stop 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- 172.21.15.28:0/493394597 shutdown_connections 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- 172.21.15.28:0/493394597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac0b4800 0x7fd9ac0ccbf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- 172.21.15.28:0/493394597 >> 172.21.15.28:0/493394597 conn(0x7fd9ac0674f0 msgr2=0x7fd9ac06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- 172.21.15.28:0/493394597 shutdown_connections 2024-03-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- 172.21.15.28:0/493394597 wait complete. 2024-03-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 Processor -- start 2024-03-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- start start 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9ac0b4420 0x7fd9ac140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac06e430 con 0x7fd9ac1f6990 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac06e2b0 con 0x7fd9ac0b4420 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.005+0000 7fd9b3000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac06e5b0 con 0x7fd9ac072190 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55828/0 (socket says 172.21.15.28:55828) 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 -- 172.21.15.28:0/432024601 learned_addr learned my addr 172.21.15.28:0/432024601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9abe00640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9a3e00640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9ac0b4420 0x7fd9ac140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 -- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 msgr2=0x7fd9ac146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 -- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9ac0b4420 msgr2=0x7fd9ac140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9ac0b4420 0x7fd9ac140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 -- 172.21.15.28:0/432024601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac1471e0 con 0x7fd9ac1f6990 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b1000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009d80 tx=0x7fd9a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4018020 con 0x7fd9ac1f6990 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4014480 con 0x7fd9ac1f6990 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4014870 con 0x7fd9ac1f6990 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac148d80 con 0x7fd9ac1f6990 2024-03-21T08:41:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac140e90 con 0x7fd9ac1f6990 2024-03-21T08:41:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac073430 con 0x7fd9ac1f6990 2024-03-21T08:41:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.009+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd9a401b020 con 0x7fd9ac1f6990 2024-03-21T08:41:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.013+0000 7fd9a8c00640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 0x7fd988069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.013+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd9a4086ee0 con 0x7fd9ac1f6990 2024-03-21T08:41:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.013+0000 7fd9abe00640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 0x7fd988069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.013+0000 7fd9abe00640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 0x7fd988069fe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007fd0 tx=0x7fd99c007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.017+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd9a4022030 con 0x7fd9ac1f6990 2024-03-21T08:41:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.181+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9ac04f4f0 con 0x7fd9ac1f6990 2024-03-21T08:41:36.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.181+0000 7fd9a8c00640 1 -- 172.21.15.28:0/432024601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9a4010040 con 0x7fd9ac1f6990 2024-03-21T08:41:36.186 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 msgr2=0x7fd988069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 0x7fd988069fe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007fd0 tx=0x7fd99c007d60 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 msgr2=0x7fd9ac1430f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009d80 tx=0x7fd9a400c490 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 shutdown_connections 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9ac1f6990 0x7fd9ac1430f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9ac0b4420 0x7fd9ac140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd988067b20 0x7fd988069fe0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 --2- 172.21.15.28:0/432024601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9ac072190 0x7fd9ac146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 >> 172.21.15.28:0/432024601 conn(0x7fd9ac0674f0 msgr2=0x7fd9ac152560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 shutdown_connections 2024-03-21T08:41:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.185+0000 7fd9b3000640 1 -- 172.21.15.28:0/432024601 wait complete. 2024-03-21T08:41:36.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 Processor -- start 2024-03-21T08:41:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 -- start start 2024-03-21T08:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb744099c90 0x7fb74409a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7440cc540 0x7fb7440cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74409b1e0 con 0x7fb74415e2a0 2024-03-21T08:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74409a630 con 0x7fb7440cc540 2024-03-21T08:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.349+0000 7fb74a600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7440d0930 con 0x7fb744099c90 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7440cc540 0x7fb7440cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7440cc540 0x7fb7440cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55830/0 (socket says 172.21.15.28:55830) 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 -- 172.21.15.28:0/2585156957 learned_addr learned my addr 172.21.15.28:0/2585156957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743400640 1 --1- 172.21.15.28:0/2585156957 >> v1:172.21.15.28:6789/0 conn(0x7fb74415e2a0 0x7fb74409aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38694/0 (socket says 172.21.15.28:38694) 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb73aa00640 1 --2- 172.21.15.28:0/2585156957 >> v2:172.21.15.151:3300/0 conn(0x7fb744099c90 0x7fb74409a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 -- 172.21.15.28:0/2585156957 >> v2:172.21.15.151:3300/0 conn(0x7fb744099c90 msgr2=0x7fb74409a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 --2- 172.21.15.28:0/2585156957 >> v2:172.21.15.151:3300/0 conn(0x7fb744099c90 0x7fb74409a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 -- 172.21.15.28:0/2585156957 >> v1:172.21.15.28:6789/0 conn(0x7fb74415e2a0 legacy=0x7fb74409aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 -- 172.21.15.28:0/2585156957 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440d0c10 con 0x7fb7440cc540 2024-03-21T08:41:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb743e00640 1 --2- 172.21.15.28:0/2585156957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7440cc540 0x7fb7440cc9d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb744072bb0 tx=0x7fb73c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619b5d84b7c5822a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb742a00640 1 -- 172.21.15.28:0/2585156957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0b2070 con 0x7fb7440cc540 2024-03-21T08:41:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb742a00640 1 -- 172.21.15.28:0/2585156957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c0ad680 con 0x7fb7440cc540 2024-03-21T08:41:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.353+0000 7fb742a00640 1 -- 172.21.15.28:0/2585156957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0ada50 con 0x7fb7440cc540 2024-03-21T08:41:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- 172.21.15.28:0/2585156957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7440cc540 msgr2=0x7fb7440cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- 172.21.15.28:0/2585156957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7440cc540 0x7fb7440cc9d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb744072bb0 tx=0x7fb73c01c490 comp rx=0 tx=0).stop 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- 172.21.15.28:0/2585156957 shutdown_connections 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- 172.21.15.28:0/2585156957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7440cc540 0x7fb7440cc9d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- 172.21.15.28:0/2585156957 >> v2:172.21.15.151:3300/0 conn(0x7fb744099c90 0x7fb74409a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- 172.21.15.28:0/2585156957 >> 172.21.15.28:0/2585156957 conn(0x7fb7440b3620 msgr2=0x7fb7440b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- 172.21.15.28:0/2585156957 shutdown_connections 2024-03-21T08:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- 172.21.15.28:0/2585156957 wait complete. 2024-03-21T08:41:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 Processor -- start 2024-03-21T08:41:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- start start 2024-03-21T08:41:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7440cc540 0x7fb744158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74415e2a0 0x7fb7441592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74415e680 con 0x7fb744099c90 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74416aee0 con 0x7fb7440cc540 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb74a600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb744154c10 con 0x7fb74415e2a0 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55834/0 (socket says 172.21.15.28:55834) 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 -- 172.21.15.28:0/1715836463 learned_addr learned my addr 172.21.15.28:0/1715836463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb73aa00640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7440cc540 0x7fb744158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743e00640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74415e2a0 0x7fb7441592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 -- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74415e2a0 msgr2=0x7fb7441592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74415e2a0 0x7fb7441592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 -- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7440cc540 msgr2=0x7fb744158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7440cc540 0x7fb744158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 -- 172.21.15.28:0/1715836463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440684e0 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.357+0000 7fb743400640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c730 tx=0x7fb73400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734017070 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73400ce40 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74415c530 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73400a800 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb744154000 con 0x7fb744099c90 2024-03-21T08:41:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74409faf0 con 0x7fb744099c90 2024-03-21T08:41:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb73401a070 con 0x7fb744099c90 2024-03-21T08:41:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 0x7fb720069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb734085810 con 0x7fb744099c90 2024-03-21T08:41:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.361+0000 7fb73aa00640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 0x7fb720069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.365+0000 7fb73aa00640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 0x7fb720069fe0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb74415d0b0 tx=0x7fb72c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.365+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb74415c530 con 0x7fb744099c90 2024-03-21T08:41:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb74404f4f0 con 0x7fb744099c90 2024-03-21T08:41:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb73be00640 1 -- 172.21.15.28:0/1715836463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb74409b1e0 con 0x7fb744099c90 2024-03-21T08:41:36.533 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 msgr2=0x7fb720069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 0x7fb720069fe0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb74415d0b0 tx=0x7fb72c009190 comp rx=0 tx=0).stop 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 msgr2=0x7fb7441538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c730 tx=0x7fb73400cbf0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 shutdown_connections 2024-03-21T08:41:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb720067b20 0x7fb720069fe0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74415e2a0 0x7fb7441592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb7440cc540 0x7fb744158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 --2- 172.21.15.28:0/1715836463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb744099c90 0x7fb7441538c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 >> 172.21.15.28:0/1715836463 conn(0x7fb7440b3620 msgr2=0x7fb74406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 shutdown_connections 2024-03-21T08:41:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.529+0000 7fb74a600640 1 -- 172.21.15.28:0/1715836463 wait complete. 2024-03-21T08:41:36.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.689+0000 7fcf74800640 1 Processor -- start 2024-03-21T08:41:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.689+0000 7fcf74800640 1 -- start start 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf74800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c15d060 0x7fcf6c09af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf74800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf6c0b1360 0x7fcf6c0b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf74800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0d0780 con 0x7fcf6c099b80 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf74800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c0cc810 con 0x7fcf6c0b1360 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf74800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0ccaf0 con 0x7fcf6c15d060 2024-03-21T08:41:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcf6c0b1360 0x7fcf6c0b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf72800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fcf6c099b80 0x7fcf6c0cd2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:51242/0 (socket says 172.21.15.28:51242) 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c15d060 0x7fcf6c09af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf72800640 1 -- 172.21.15.28:0/4217216695 learned_addr learned my addr 172.21.15.28:0/4217216695 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf70a00640 1 -- 172.21.15.28:0/4217216695 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232961102 0 0) 0x7fcf6c0d0780 con 0x7fcf6c099b80 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71400640 1 -- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c15d060 msgr2=0x7fcf6c09af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71400640 1 --2- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c15d060 0x7fcf6c09af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71400640 1 -- 172.21.15.28:0/4217216695 >> v1:172.21.15.151:6789/0 conn(0x7fcf6c099b80 legacy=0x7fcf6c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.693+0000 7fcf71400640 1 -- 172.21.15.28:0/4217216695 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0ccdd0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf71400640 1 --2- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c0b1360 0x7fcf6c0b1780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0097c0 tx=0x7fcf5c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbd13ad3db984b20 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf70a00640 1 -- 172.21.15.28:0/4217216695 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0aa620 con 0x7fcf6c0b1360 2024-03-21T08:41:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf70a00640 1 -- 172.21.15.28:0/4217216695 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c0af070 con 0x7fcf6c0b1360 2024-03-21T08:41:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf70a00640 1 -- 172.21.15.28:0/4217216695 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0aa9e0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c0b1360 msgr2=0x7fcf6c0b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c0b1360 0x7fcf6c0b1780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0097c0 tx=0x7fcf5c01c460 comp rx=0 tx=0).stop 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- 172.21.15.28:0/4217216695 shutdown_connections 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c0b1360 0x7fcf6c0b1780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- 172.21.15.28:0/4217216695 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c15d060 0x7fcf6c09af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- 172.21.15.28:0/4217216695 >> 172.21.15.28:0/4217216695 conn(0x7fcf6c0b3510 msgr2=0x7fcf6c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- 172.21.15.28:0/4217216695 shutdown_connections 2024-03-21T08:41:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- 172.21.15.28:0/4217216695 wait complete. 2024-03-21T08:41:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 Processor -- start 2024-03-21T08:41:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- start start 2024-03-21T08:41:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c099b80 0x7fcf6c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 0x7fcf6c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 0x7fcf6c156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0b0b70 con 0x7fcf6c15d060 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0b09f0 con 0x7fcf6c099b80 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf74800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0d0310 con 0x7fcf6c0b1360 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.697+0000 7fcf72800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 0x7fcf6c156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c099b80 0x7fcf6c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf72800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 0x7fcf6c156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55844/0 (socket says 172.21.15.28:55844) 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf72800640 1 -- 172.21.15.28:0/3797749121 learned_addr learned my addr 172.21.15.28:0/3797749121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 0x7fcf6c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 -- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c099b80 msgr2=0x7fcf6c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c099b80 0x7fcf6c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 -- 172.21.15.28:0/3797749121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 msgr2=0x7fcf6c156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 0x7fcf6c156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 -- 172.21.15.28:0/3797749121 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c151f80 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf71400640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 0x7fcf6c154630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c073320 tx=0x7fcf5c01aca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0aabe0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c0aadc0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0be650 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c0cf0d0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c152ad0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c04f410 con 0x7fcf6c0b1360 2024-03-21T08:41:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcf5c0bc020 con 0x7fcf6c0b1360 2024-03-21T08:41:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 0x7fcf50069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.701+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcf5c120070 con 0x7fcf6c0b1360 2024-03-21T08:41:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.705+0000 7fcf71e00640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 0x7fcf50069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.705+0000 7fcf71e00640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 0x7fcf50069fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c15a680 tx=0x7fcf60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:36.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.709+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcf5c0c5030 con 0x7fcf6c0b1360 2024-03-21T08:41:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.905+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf6c1569c0 con 0x7fcf6c0b1360 2024-03-21T08:41:36.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.905+0000 7fcf66000640 1 -- 172.21.15.28:0/3797749121 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7fcf5c016050 con 0x7fcf6c0b1360 2024-03-21T08:41:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:36.911 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 msgr2=0x7fcf50069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 0x7fcf50069fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c15a680 tx=0x7fcf60009190 comp rx=0 tx=0).stop 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 msgr2=0x7fcf6c154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 0x7fcf6c154630 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c073320 tx=0x7fcf5c01aca0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 shutdown_connections 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c15d060 0x7fcf6c156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcf50067b20 0x7fcf50069fe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcf6c0b1360 0x7fcf6c154630 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 --2- 172.21.15.28:0/3797749121 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcf6c099b80 0x7fcf6c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 >> 172.21.15.28:0/3797749121 conn(0x7fcf6c0b3510 msgr2=0x7fcf6c15ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 shutdown_connections 2024-03-21T08:41:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:36.909+0000 7fcf74800640 1 -- 172.21.15.28:0/3797749121 wait complete. 2024-03-21T08:41:36.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 Processor -- start 2024-03-21T08:41:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 -- start start 2024-03-21T08:41:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3f4099b80 0x7ff3f4099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40cdb50 con 0x7ff3f40c5f80 2024-03-21T08:41:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40cde30 con 0x7ff3f415f260 2024-03-21T08:41:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f409a520 con 0x7ff3f4099b80 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3e9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3f4099b80 0x7ff3f4099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f1e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff3f415f260 0x7ff3f40cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:51252/0 (socket says 172.21.15.28:51252) 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3f40c5f80 0x7ff3f40c6410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38696/0 (socket says 172.21.15.28:38696) 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3f1e00640 1 -- 172.21.15.28:0/1282571283 learned_addr learned my addr 172.21.15.28:0/1282571283 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3e9400640 1 -- 172.21.15.28:0/1282571283 >> v1:172.21.15.151:6789/0 conn(0x7ff3f415f260 legacy=0x7ff3f40cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3e9400640 1 -- 172.21.15.28:0/1282571283 >> v1:172.21.15.28:6789/0 conn(0x7ff3f40c5f80 legacy=0x7ff3f40c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.069+0000 7ff3e9400640 1 -- 172.21.15.28:0/1282571283 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40c5780 con 0x7ff3f4099b80 2024-03-21T08:41:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3e9400640 1 --2- 172.21.15.28:0/1282571283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f4099b80 0x7ff3f4099fe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009d90 tx=0x7ff3dc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27d7022c8a71765 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f1400640 1 -- 172.21.15.28:0/1282571283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0b2070 con 0x7ff3f4099b80 2024-03-21T08:41:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f1400640 1 -- 172.21.15.28:0/1282571283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc0a58b0 con 0x7ff3f4099b80 2024-03-21T08:41:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f1400640 1 -- 172.21.15.28:0/1282571283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0a5c80 con 0x7ff3f4099b80 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1282571283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f4099b80 msgr2=0x7ff3f4099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1282571283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f4099b80 0x7ff3f4099fe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009d90 tx=0x7ff3dc01c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1282571283 shutdown_connections 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1282571283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f4099b80 0x7ff3f4099fe0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1282571283 >> 172.21.15.28:0/1282571283 conn(0x7ff3f40b34d0 msgr2=0x7ff3f40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1282571283 shutdown_connections 2024-03-21T08:41:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.073+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1282571283 wait complete. 2024-03-21T08:41:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 Processor -- start 2024-03-21T08:41:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- start start 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3f4099b80 0x7ff3f4152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 0x7ff3f40b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f415f7c0 con 0x7ff3f40c5f80 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f415f640 con 0x7ff3f4099b80 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4169a50 con 0x7ff3f415f260 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55864/0 (socket says 172.21.15.28:55864) 2024-03-21T08:41:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 -- 172.21.15.28:0/1661522754 learned_addr learned my addr 172.21.15.28:0/1661522754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f1e00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3f4099b80 0x7ff3f4152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f2800640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 0x7ff3f40b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 -- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 msgr2=0x7ff3f40b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 0x7ff3f40b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 -- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3f4099b80 msgr2=0x7ff3f4152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3f4099b80 0x7ff3f4152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 -- 172.21.15.28:0/1661522754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4153430 con 0x7ff3f40c5f80 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f2800640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 0x7ff3f40b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3e9400640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc0b0040 tx=0x7ff3dc019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0b2050 con 0x7ff3f40c5f80 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc0be070 con 0x7ff3f40c5f80 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0adb10 con 0x7ff3f40c5f80 2024-03-21T08:41:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f415e5c0 con 0x7ff3f40c5f80 2024-03-21T08:41:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.077+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f406ce30 con 0x7ff3f40c5f80 2024-03-21T08:41:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f404f410 con 0x7ff3f40c5f80 2024-03-21T08:41:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff3dc0adcb0 con 0x7ff3f40c5f80 2024-03-21T08:41:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 0x7ff3d0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff3dc121330 con 0x7ff3f40c5f80 2024-03-21T08:41:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3f1e00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 0x7ff3d0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.081+0000 7ff3f1e00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 0x7ff3d0069fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4005890 tx=0x7ff3e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.085+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff3dc016560 con 0x7ff3f40c5f80 2024-03-21T08:41:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f40c51e0 con 0x7ff3f40c5f80 2024-03-21T08:41:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3eaa00640 1 -- 172.21.15.28:0/1661522754 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7ff3dc0f1a90 con 0x7ff3f40c5f80 2024-03-21T08:41:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:37.257 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 msgr2=0x7ff3d0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 0x7ff3d0069fe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4005890 tx=0x7ff3e4009040 comp rx=0 tx=0).stop 2024-03-21T08:41:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 msgr2=0x7ff3f406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc0b0040 tx=0x7ff3dc019680 comp rx=0 tx=0).stop 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 shutdown_connections 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff3d0067b20 0x7ff3d0069fe0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff3f415f260 0x7ff3f40b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3f40c5f80 0x7ff3f406d350 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 --2- 172.21.15.28:0/1661522754 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff3f4099b80 0x7ff3f4152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 >> 172.21.15.28:0/1661522754 conn(0x7ff3f40b34d0 msgr2=0x7ff3f40cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 shutdown_connections 2024-03-21T08:41:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.253+0000 7ff3f8c00640 1 -- 172.21.15.28:0/1661522754 wait complete. 2024-03-21T08:41:37.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:41:37.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 Processor -- start 2024-03-21T08:41:37.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 -- start start 2024-03-21T08:41:37.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97d80b5520 0x7f97d80cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d815bf00 con 0x7f97d80b34f0 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d80cdad0 con 0x7f97d80af060 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97dda00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d80ce6e0 con 0x7f97d80b5520 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97d80b5520 0x7f97d80cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97d80b34f0 0x7f97d815c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38712/0 (socket says 172.21.15.28:38712) 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d7400640 1 -- 172.21.15.28:0/835653077 learned_addr learned my addr 172.21.15.28:0/835653077 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d6000640 1 -- 172.21.15.28:0/835653077 >> v1:172.21.15.151:6789/0 conn(0x7f97d80af060 legacy=0x7f97d80b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d6000640 1 -- 172.21.15.28:0/835653077 >> v1:172.21.15.28:6789/0 conn(0x7f97d80b34f0 legacy=0x7f97d815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.417+0000 7f97d6000640 1 -- 172.21.15.28:0/835653077 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d80cddb0 con 0x7f97d80b5520 2024-03-21T08:41:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97d6000640 1 --2- 172.21.15.28:0/835653077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d80b5520 0x7f97d80cd380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ce860 tx=0x7f97c401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9e03ea8bb1e728 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97d5600640 1 -- 172.21.15.28:0/835653077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40b0070 con 0x7f97d80b5520 2024-03-21T08:41:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97d5600640 1 -- 172.21.15.28:0/835653077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c40ab600 con 0x7f97d80b5520 2024-03-21T08:41:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97d5600640 1 -- 172.21.15.28:0/835653077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40ab9d0 con 0x7f97d80b5520 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- 172.21.15.28:0/835653077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d80b5520 msgr2=0x7f97d80cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 --2- 172.21.15.28:0/835653077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d80b5520 0x7f97d80cd380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ce860 tx=0x7f97c401c410 comp rx=0 tx=0).stop 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- 172.21.15.28:0/835653077 shutdown_connections 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 --2- 172.21.15.28:0/835653077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d80b5520 0x7f97d80cd380 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- 172.21.15.28:0/835653077 >> 172.21.15.28:0/835653077 conn(0x7f97d80cc230 msgr2=0x7f97d80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:37.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- 172.21.15.28:0/835653077 shutdown_connections 2024-03-21T08:41:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- 172.21.15.28:0/835653077 wait complete. 2024-03-21T08:41:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 Processor -- start 2024-03-21T08:41:37.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.421+0000 7f97dda00640 1 -- start start 2024-03-21T08:41:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97d80b34f0 0x7f97d81362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97d80b5520 0x7f97d815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d815d2a0 con 0x7f97d815ebf0 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d815d120 con 0x7f97d80b34f0 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d815d420 con 0x7f97d80b5520 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55876/0 (socket says 172.21.15.28:55876) 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 -- 172.21.15.28:0/1576791141 learned_addr learned my addr 172.21.15.28:0/1576791141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d6a00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97d80b34f0 0x7f97d81362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d6000640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97d80b5520 0x7f97d815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 -- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97d80b5520 msgr2=0x7f97d815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97d80b5520 0x7f97d815e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 -- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97d80b34f0 msgr2=0x7f97d81362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97d80b34f0 0x7f97d81362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 -- 172.21.15.28:0/1576791141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8137a00 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97d7400640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f97d000cbc0 tx=0x7f97d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0017070 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d000ad40 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012600 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d8137ce0 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d81382a0 con 0x7f97d815ebf0 2024-03-21T08:41:37.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.425+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d804f410 con 0x7f97d815ebf0 2024-03-21T08:41:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.429+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f97d001a070 con 0x7f97d815ebf0 2024-03-21T08:41:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.429+0000 7f97cea00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 0x7f97b806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.429+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f97d0085f50 con 0x7f97d815ebf0 2024-03-21T08:41:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.429+0000 7f97d6a00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 0x7f97b806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.429+0000 7f97d6a00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 0x7f97b806a030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f97c8007fd0 tx=0x7f97c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:37.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.433+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f97d0055fb0 con 0x7f97d815ebf0 2024-03-21T08:41:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.629+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97d8072680 con 0x7f97d815ebf0 2024-03-21T08:41:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.629+0000 7f97cea00640 1 -- 172.21.15.28:0/1576791141 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f97d005a050 con 0x7f97d815ebf0 2024-03-21T08:41:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:37.635 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:41:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 msgr2=0x7f97b806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 0x7f97b806a030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f97c8007fd0 tx=0x7f97c8009040 comp rx=0 tx=0).stop 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 msgr2=0x7f97d8136c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f97d000cbc0 tx=0x7f97d000a430 comp rx=0 tx=0).stop 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 shutdown_connections 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d815ebf0 0x7f97d8136c90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97b8067b70 0x7f97b806a030 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97d80b5520 0x7f97d815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 --2- 172.21.15.28:0/1576791141 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97d80b34f0 0x7f97d81362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 >> 172.21.15.28:0/1576791141 conn(0x7f97d80cc230 msgr2=0x7f97d8073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 shutdown_connections 2024-03-21T08:41:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:37.633+0000 7f97dda00640 1 -- 172.21.15.28:0/1576791141 wait complete. 2024-03-21T08:41:37.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:41:37.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:41:38.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-21T08:41:07.397365+0000', 'modified': '2024-03-21T08:41:25.469098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.151:6836/4096075958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6834', 'nonce': 4096075958}, {'type': 'v1', 'addr': '172.21.15.151:6836', 'nonce': 4096075958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-03-21T08:41:38.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-21T08:41:38.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-21T08:41:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 Processor -- start 2024-03-21T08:41:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 -- start start 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f223006f0a0 0x7f22300cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f223015cbb0 0x7f22300cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223006f500 con 0x7f2230071a80 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22300ce380 con 0x7f223015cbb0 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2235600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223015c720 con 0x7f223006f0a0 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f223015cbb0 0x7f22300cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f223015cbb0 0x7f22300cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55888/0 (socket says 172.21.15.28:55888) 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f2227e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f223006f0a0 0x7f22300cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 -- 172.21.15.28:0/960583622 learned_addr learned my addr 172.21.15.28:0/960583622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222ea00640 1 --1- 172.21.15.28:0/960583622 >> v1:172.21.15.151:6789/0 conn(0x7f2230071a80 0x7f2230071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:51278/0 (socket says 172.21.15.28:51278) 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 -- 172.21.15.28:0/960583622 >> v2:172.21.15.151:3300/0 conn(0x7f223006f0a0 msgr2=0x7f22300cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 --2- 172.21.15.28:0/960583622 >> v2:172.21.15.151:3300/0 conn(0x7f223006f0a0 0x7f22300cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 -- 172.21.15.28:0/960583622 >> v1:172.21.15.151:6789/0 conn(0x7f2230071a80 legacy=0x7f2230071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.201+0000 7f222f400640 1 -- 172.21.15.28:0/960583622 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22300ce5b0 con 0x7f223015cbb0 2024-03-21T08:41:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f222f400640 1 --2- 172.21.15.28:0/960583622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223015cbb0 0x7f22300cc4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f22300af710 tx=0x7f222801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9181230b2648d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f222e000640 1 -- 172.21.15.28:0/960583622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280a57f0 con 0x7f223015cbb0 2024-03-21T08:41:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f222e000640 1 -- 172.21.15.28:0/960583622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22280a5990 con 0x7f223015cbb0 2024-03-21T08:41:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f222e000640 1 -- 172.21.15.28:0/960583622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280a5d60 con 0x7f223015cbb0 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- 172.21.15.28:0/960583622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223015cbb0 msgr2=0x7f22300cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 --2- 172.21.15.28:0/960583622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223015cbb0 0x7f22300cc4b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f22300af710 tx=0x7f222801c930 comp rx=0 tx=0).stop 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- 172.21.15.28:0/960583622 shutdown_connections 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 --2- 172.21.15.28:0/960583622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223015cbb0 0x7f22300cc4b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 --2- 172.21.15.28:0/960583622 >> v2:172.21.15.151:3300/0 conn(0x7f223006f0a0 0x7f22300cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- 172.21.15.28:0/960583622 >> 172.21.15.28:0/960583622 conn(0x7f22300674f0 msgr2=0x7f223006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- 172.21.15.28:0/960583622 shutdown_connections 2024-03-21T08:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- 172.21.15.28:0/960583622 wait complete. 2024-03-21T08:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 Processor -- start 2024-03-21T08:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.205+0000 7f2235600640 1 -- start start 2024-03-21T08:41:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 0x7f223013fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 0x7f22301401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230144b20 con 0x7f223006f0a0 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300cce00 con 0x7f2230071a80 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2230168f20 con 0x7f223015cbb0 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55892/0 (socket says 172.21.15.28:55892) 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 -- 172.21.15.28:0/946990234 learned_addr learned my addr 172.21.15.28:0/946990234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2227e00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 0x7f223013fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222f400640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 0x7f22301401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 -- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 msgr2=0x7f22301401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 0x7f22301401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 -- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 msgr2=0x7f223013fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 0x7f223013fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 -- 172.21.15.28:0/946990234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22300a8df0 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222f400640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 0x7f22301401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2227e00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 0x7f223013fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f222ea00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f222000c420 tx=0x7f222000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2226a00640 1 -- 172.21.15.28:0/946990234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200074e0 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2226a00640 1 -- 172.21.15.28:0/946990234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2220007680 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2230140800 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2226a00640 1 -- 172.21.15.28:0/946990234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220004a30 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2230140dc0 con 0x7f223006f0a0 2024-03-21T08:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.209+0000 7f2226a00640 1 -- 172.21.15.28:0/946990234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f222001a020 con 0x7f223006f0a0 2024-03-21T08:41:38.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2226a00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 0x7f220c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2226a00640 1 -- 172.21.15.28:0/946990234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2220085ac0 con 0x7f223006f0a0 2024-03-21T08:41:38.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 0x7f220c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 Processor -- start 2024-03-21T08:41:38.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 -- start start 2024-03-21T08:41:38.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 0x7f220c069fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f223013e780 tx=0x7f2218008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc2477d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc242e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc227b680 con 0x560cc2478240 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc2202b00 con 0x560cc242dff0 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560cc2277720 con 0x560cc2477920 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55898/0 (socket says 172.21.15.28:55898) 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 -- 172.21.15.28:0/2195113092 learned_addr learned my addr 172.21.15.28:0/2195113092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f222ea00640 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc2477d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f222f400640 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc242e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 -- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 msgr2=0x560cc2477d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc2477d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 -- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 msgr2=0x560cc242e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc242e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 -- 172.21.15.28:0/2195113092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cc2253940 con 0x560cc2478240 2024-03-21T08:41:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.213+0000 7f2227e00640 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2218011d90 tx=0x7f2218014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cd9c539a7144ff server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f221f400640 1 -- 172.21.15.28:0/2195113092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218027070 con 0x560cc2478240 2024-03-21T08:41:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f221f400640 1 -- 172.21.15.28:0/2195113092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221801d050 con 0x560cc2478240 2024-03-21T08:41:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f221f400640 1 -- 172.21.15.28:0/2195113092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180098e0 con 0x560cc2478240 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- 172.21.15.28:0/2195113092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 msgr2=0x560cc242dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2218011d90 tx=0x7f2218014000 comp rx=0 tx=0).stop 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- 172.21.15.28:0/2195113092 shutdown_connections 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc242e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242dab0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- 172.21.15.28:0/2195113092 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc2477d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- 172.21.15.28:0/2195113092 >> 172.21.15.28:0/2195113092 conn(0x560cc2429fa0 msgr2=0x560cc242c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- 172.21.15.28:0/2195113092 shutdown_connections 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- 172.21.15.28:0/2195113092 wait complete. 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- start start 2024-03-21T08:41:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 Processor -- start 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc2479670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2478240 0x560cc247a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc227a990 con 0x560cc2477920 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc2235ae0 con 0x560cc242dff0 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560cc2213750 con 0x560cc2478240 2024-03-21T08:41:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2238b82000 10 client.0 _ll_register_callbacks cb 0x560cc2418ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55912/0 (socket says 172.21.15.28:55912) 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 -- 172.21.15.28:0/3343033116 learned_addr learned my addr 172.21.15.28:0/3343033116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f222ea00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc2479670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f222f400640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2478240 0x560cc247a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 -- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2478240 msgr2=0x560cc247a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2478240 0x560cc247a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 -- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 msgr2=0x560cc2479670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc2479670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 -- 172.21.15.28:0/3343033116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cc22c0d80 con 0x560cc2477920 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2227e00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22180039f0 tx=0x7f22180046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af0065b9ca347e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.217+0000 7f2213e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:41:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218032020 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f221801d050 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2218015710 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2218027070 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 10 client.4842 fetch_fsmap learned FSMap version 9 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x560cc2202d10 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221801d050 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f2218015910 con 0x560cc2477920 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 10 client.4842 fetch_fsmap finished waiting for FSMap version 9 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 10 client.4842 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 20 client.4842 populate_metadata read hostname 'smithi028' 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc247ab00 0x560cc248e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 4 client.4842 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 --> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cc2216b60 con 0x560cc247ab00 2024-03-21T08:41:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f222ea00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc247ab00 0x560cc248e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f222ea00640 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc247ab00 0x560cc248e2d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e66ebb57c5e3ac4b server_cookie=67f344a80875836f in_seq=0 out_seq=0 2024-03-21T08:41:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 10 client.4842 ms_handle_connect on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 1 -- 172.21.15.28:0/3343033116 <== mds.0 v2:172.21.15.151:6834/4096075958 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560cc2216b60 con 0x560cc247ab00 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 10 client.4842 handle_command_reply: tid=0 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 client.4842 shutdown 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 msgr2=0x560cc2479bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22180039f0 tx=0x7f22180046d0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 shutdown_connections 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc247ab00 0x560cc248e2d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2478240 0x560cc247a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2477920 0x560cc2479bb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 --2- 172.21.15.28:0/3343033116 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc242dff0 0x560cc2479670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2213e00640 0 client.4842 ms_handle_reset on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 >> 172.21.15.28:0/3343033116 conn(0x560cc2429fa0 msgr2=0x560cc242cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 shutdown_connections 2024-03-21T08:41:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 1 -- 172.21.15.28:0/3343033116 wait complete. 2024-03-21T08:41:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.221+0000 7f2238b82000 20 client.4842 trim_cache size 0 max 16384 2024-03-21T08:41:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.237+0000 7f2238b82000 1 Processor -- start 2024-03-21T08:41:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.237+0000 7f2238b82000 1 -- start start 2024-03-21T08:41:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc242dff0 0x560cc247df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2477920 0x560cc247e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22301ff750 con 0x560cc2478240 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300b4910 con 0x560cc2477920 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22301f5bb0 con 0x560cc242dff0 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55914/0 (socket says 172.21.15.28:55914) 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 -- 172.21.15.28:0/408911247 learned_addr learned my addr 172.21.15.28:0/408911247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2227e00640 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2477920 0x560cc247e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222ea00640 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc242dff0 0x560cc247df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 -- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc242dff0 msgr2=0x560cc247df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc242dff0 0x560cc247df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 -- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2477920 msgr2=0x560cc247e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2477920 0x560cc247e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 -- 172.21.15.28:0/408911247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cc2248540 con 0x560cc2478240 2024-03-21T08:41:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f222f400640 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22280af040 tx=0x7f22280a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbaa8cd92590454 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f221ea00640 1 -- 172.21.15.28:0/408911247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280a57f0 con 0x560cc2478240 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f221ea00640 1 -- 172.21.15.28:0/408911247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22280c5070 con 0x560cc2478240 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f221ea00640 1 -- 172.21.15.28:0/408911247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280a5b90 con 0x560cc2478240 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- 172.21.15.28:0/408911247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 msgr2=0x560cc242c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f22280af040 tx=0x7f22280a7070 comp rx=0 tx=0).stop 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- 172.21.15.28:0/408911247 shutdown_connections 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc2478240 0x560cc242c390 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2477920 0x560cc247e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- 172.21.15.28:0/408911247 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc242dff0 0x560cc247df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- 172.21.15.28:0/408911247 >> 172.21.15.28:0/408911247 conn(0x560cc2429fa0 msgr2=0x560cc24895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- 172.21.15.28:0/408911247 shutdown_connections 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- 172.21.15.28:0/408911247 wait complete. 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- start start 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 Processor -- start 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc247e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 0x560cc2484f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc2338700 con 0x560cc242dff0 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cc2223a80 con 0x560cc2478240 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.241+0000 7f2238b82000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x560cc2429e20 con 0x560cc2477920 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 10 client.0 _ll_register_callbacks cb 0x7f2218025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55930/0 (socket says 172.21.15.28:55930) 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2227e00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc247e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 -- 172.21.15.28:0/3732670004 learned_addr learned my addr 172.21.15.28:0/3732670004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222f400640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 0x560cc2484f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 -- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 msgr2=0x560cc247e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc247e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 -- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 msgr2=0x560cc2484f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 0x560cc2484f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 -- 172.21.15.28:0/3732670004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cc220f4b0 con 0x560cc242dff0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222f400640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 0x560cc2484f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2227e00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc247e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f222ea00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2220051ee0 tx=0x7f2220000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfff41d75e7dc50c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220019da0 con 0x560cc242dff0 2024-03-21T08:41:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f22200520f0 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22280c5070 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f222007b050 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 10 client.4848 fetch_fsmap learned FSMap version 9 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x560cc22222f0 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22280c5070 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f2220055ce0 con 0x560cc242dff0 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 10 client.4848 fetch_fsmap finished waiting for FSMap version 9 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 10 client.4848 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 20 client.4848 populate_metadata read hostname 'smithi028' 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc24e7600 0x560cc24e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 4 client.4848 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 --> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560cc2288bc0 con 0x560cc24e7600 2024-03-21T08:41:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2227e00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc24e7600 0x560cc24e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2227e00640 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc24e7600 0x560cc24e99f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d169799581badf3 server_cookie=63c4e02f15a2892b in_seq=0 out_seq=0 2024-03-21T08:41:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 10 client.4848 ms_handle_connect on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 1 -- 172.21.15.28:0/3732670004 <== mds.0 v2:172.21.15.151:6834/4096075958 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x560cc2288bc0 con 0x560cc24e7600 2024-03-21T08:41:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 10 client.4848 handle_command_reply: tid=0 2024-03-21T08:41:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 client.4848 shutdown 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 msgr2=0x560cc247df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2220051ee0 tx=0x7f2220000ee0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 shutdown_connections 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:6834/4096075958,v1:172.21.15.151:6836/4096075958] conn(0x560cc24e7600 0x560cc24e99f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x560cc2478240 0x560cc2484f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x560cc2477920 0x560cc247e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 --2- 172.21.15.28:0/3732670004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cc242dff0 0x560cc247df40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2213e00640 0 client.4848 ms_handle_reset on v2:172.21.15.151:6834/4096075958 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 >> 172.21.15.28:0/3732670004 conn(0x560cc2429fa0 msgr2=0x560cc24759d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 shutdown_connections 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 1 -- 172.21.15.28:0/3732670004 wait complete. 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2238b82000 20 client.4848 trim_cache size 0 max 16384 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "51dba44a-cba1-4b57-a836-26a407968834", 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4293, 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-21T08:41:38.256 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 30.252711048999998, 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 9, 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 25, 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 25, 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 37.283459137999998 2024-03-21T08:41:38.257 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 msgr2=0x7f220c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 0x7f220c069fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f223013e780 tx=0x7f2218008040 comp rx=0 tx=0).stop 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 msgr2=0x7f223013a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f222000c420 tx=0x7f222000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 shutdown_connections 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f220c067b20 0x7f220c069fe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.245+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f223015cbb0 0x7f22301401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.249+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2230071a80 0x7f223013fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.249+0000 7f2235600640 1 --2- 172.21.15.28:0/946990234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223006f0a0 0x7f223013a1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.249+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 >> 172.21.15.28:0/946990234 conn(0x7f22300674f0 msgr2=0x7f223013ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.249+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 shutdown_connections 2024-03-21T08:41:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.249+0000 7f2235600640 1 -- 172.21.15.28:0/946990234 wait complete. 2024-03-21T08:41:38.271 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-21T08:41:38.283 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-21T08:41:38.283 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-21T08:41:38.284 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:38.284 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:38.284 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:38.284 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-21T08:41:38.284 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:38.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:38.285 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-03-21T08:41:38.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:38.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link delete ceph-brx 2024-03-21T08:41:38.359 INFO:teuthology.orchestra.run.smithi151.stderr:Cannot find device "ceph-brx" 2024-03-21T08:41:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:41:38.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:38.362 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:41:38.372 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-21T08:41:38.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:38.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-21T08:41:38.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:38.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-03-21T08:41:38.650 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-21T08:41:38.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:41:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.785+0000 7f80a8800640 1 Processor -- start 2024-03-21T08:41:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.785+0000 7f80a8800640 1 -- start start 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a8800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 0x7f80a015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a8800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f80a00b4380 0x7f80a00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a8800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a015ebf0 con 0x7f80a00ce490 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a8800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a015eed0 con 0x7f80a00b4380 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a8800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a015f1b0 con 0x7f80a015e610 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80a00ce490 0x7f80a00ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38726/0 (socket says 172.21.15.28:38726) 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a5e00640 1 -- 172.21.15.28:0/2256158304 learned_addr learned my addr 172.21.15.28:0/2256158304 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a6800640 1 --2- 172.21.15.28:0/2256158304 >> v2:172.21.15.151:3300/0 conn(0x7f80a00b4380 0x7f80a00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a5400640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 0x7f80a015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a4a00640 1 -- 172.21.15.28:0/2256158304 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654343198 0 0) 0x7f80a015ebf0 con 0x7f80a00ce490 2024-03-21T08:41:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a4a00640 1 -- 172.21.15.28:0/2256158304 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088003530 con 0x7f80a00ce490 2024-03-21T08:41:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a6800640 1 -- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 msgr2=0x7f80a015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a6800640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 0x7f80a015ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a6800640 1 -- 172.21.15.28:0/2256158304 >> v1:172.21.15.28:6789/0 conn(0x7f80a00ce490 legacy=0x7f80a00ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a6800640 1 -- 172.21.15.28:0/2256158304 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a015f490 con 0x7f80a00b4380 2024-03-21T08:41:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.789+0000 7f80a5400640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 0x7f80a015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a6800640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a00b4380 0x7f80a00b4810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f809c009d90 tx=0x7f809c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ea830e3edc12fe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a4a00640 1 -- 172.21.15.28:0/2256158304 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c0b2070 con 0x7f80a00b4380 2024-03-21T08:41:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a4a00640 1 -- 172.21.15.28:0/2256158304 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c0a57a0 con 0x7f80a00b4380 2024-03-21T08:41:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a4a00640 1 -- 172.21.15.28:0/2256158304 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c0a5b70 con 0x7f80a00b4380 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a00b4380 msgr2=0x7f80a00b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a00b4380 0x7f80a00b4810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f809c009d90 tx=0x7f809c01c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- 172.21.15.28:0/2256158304 shutdown_connections 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a00b4380 0x7f80a00b4810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 --2- 172.21.15.28:0/2256158304 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a015e610 0x7f80a015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- 172.21.15.28:0/2256158304 >> 172.21.15.28:0/2256158304 conn(0x7f80a00cc230 msgr2=0x7f80a00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- 172.21.15.28:0/2256158304 shutdown_connections 2024-03-21T08:41:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- 172.21.15.28:0/2256158304 wait complete. 2024-03-21T08:41:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 Processor -- start 2024-03-21T08:41:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.793+0000 7f80a8800640 1 -- start start 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 0x7f80a0150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a015e610 0x7f80a00a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0167d40 con 0x7f80a00b4380 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0167bc0 con 0x7f80a015e610 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0167ec0 con 0x7f80a00ce490 2024-03-21T08:41:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55942/0 (socket says 172.21.15.28:55942) 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 -- 172.21.15.28:0/2851565660 learned_addr learned my addr 172.21.15.28:0/2851565660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5400640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 0x7f80a0150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a6800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a015e610 0x7f80a00a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 -- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 msgr2=0x7f80a0150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 0x7f80a0150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 -- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a015e610 msgr2=0x7f80a00a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a015e610 0x7f80a00a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 -- 172.21.15.28:0/2851565660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0151a00 con 0x7f80a00b4380 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5400640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 0x7f80a0150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a5e00640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f809400c450 tx=0x7f809400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094017070 con 0x7f80a00b4380 2024-03-21T08:41:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8094005cf0 con 0x7f80a00b4380 2024-03-21T08:41:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a00aa820 con 0x7f80a00b4380 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094007290 con 0x7f80a00b4380 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a0151ca0 con 0x7f80a00b4380 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.797+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a004f410 con 0x7f80a00b4380 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.801+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f809401a070 con 0x7f80a00b4380 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.801+0000 7f809a000640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 0x7f8084069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.801+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f80940855f0 con 0x7f80a00b4380 2024-03-21T08:41:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.801+0000 7f80a5400640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 0x7f8084069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.801+0000 7f80a5400640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 0x7f8084069fe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f80a00ae4c0 tx=0x7f8090009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:38.805+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f80940555d0 con 0x7f80a00b4380 2024-03-21T08:41:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f80a0072c10 con 0x7f80a00b4380 2024-03-21T08:41:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f809a000640 1 -- 172.21.15.28:0/2851565660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f8094059670 con 0x7f80a00b4380 2024-03-21T08:41:39.008 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 msgr2=0x7f8084069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 0x7f8084069fe0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f80a00ae4c0 tx=0x7f8090009190 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 msgr2=0x7f80a01524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f809400c450 tx=0x7f809400c910 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 shutdown_connections 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80a015e610 0x7f80a00a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8084067b20 0x7f8084069fe0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80a00ce490 0x7f80a0150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 --2- 172.21.15.28:0/2851565660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a00b4380 0x7f80a01524a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 >> 172.21.15.28:0/2851565660 conn(0x7f80a00cc230 msgr2=0x7f80a00b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 shutdown_connections 2024-03-21T08:41:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:39.005+0000 7f80a8800640 1 -- 172.21.15.28:0/2851565660 wait complete. 2024-03-21T08:41:39.029 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:41:39.029 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:41:39.029 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:41:39.029 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:41:39.029 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:39.030 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:41:39.030 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:41:39.030 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:41:39.030 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:41:39.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.030 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:41:39.039 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:39.039 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:39.039 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:39.039 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:39.039 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42371sec preferred_lft 42371sec 2024-03-21T08:41:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:41:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:39.042 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-21T08:41:39.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.042 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:39.042 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:39.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-03-21T08:41:39.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-03-21T08:41:39.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-03-21T08:41:39.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-21T08:41:39.043 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:39.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.157 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:41:39.175 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:41:39.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.177 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:41:39.233 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:41:39.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:41:39.234 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:39.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.326 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:41:39.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.335 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-03-21T08:41:39.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.388 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:39.388 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:39.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:39.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-21T08:41:39.388 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:39.473 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-21T08:41:39.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-21T08:41:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-21T08:41:39.474 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:39.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.614 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:39.614 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:39.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:41:39.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-03-21T08:41:39.614 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:39.653 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:41:39.654 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:39.654 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:39.704 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:41:39.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:39.704 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:39.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:39.816 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:39.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:39.876 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:39.927 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:41:39.928 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:41:39.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:39.995 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:40.054 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:40.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.056 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:40.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:41:40.065+0000 7f3271c64580 -1 init, newargv = 0x55ff74e25560 newargc=15 2024-03-21T08:41:40.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[34009]: starting ceph client 2024-03-21T08:41:40.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[34009]: starting fuse 2024-03-21T08:41:40.080 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:41:40.081 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:41:40.081 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:40.082 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:41:40.083 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:41:40.210 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.34009.asok 2024-03-21T08:41:40.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:41:40.221 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34009.asok 2024-03-21T08:41:40.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34009.asok status 2024-03-21T08:41:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "34009", 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:41:40.400 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4863, 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4863 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1486461826 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1486461826 2024-03-21T08:41:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4863 192.168.0.1:0/1486461826", 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1486461826", 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 9, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 25, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:41:40.402 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:41:40.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:41:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.549+0000 7ff1f3400640 1 Processor -- start 2024-03-21T08:41:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.549+0000 7ff1f3400640 1 -- start start 2024-03-21T08:41:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 0x7ff1ec0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f3400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 0x7ff1ec15c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f3400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec15bf00 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f3400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0ce6e0 con 0x7ff1ec0b5520 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f3400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec0cdad0 con 0x7ff1ec0af060 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff1ec0b34f0 0x7ff1ec0b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56892/0 (socket says 172.21.15.28:56892) 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1e3e00640 1 -- 172.21.15.28:0/3132163674 learned_addr learned my addr 172.21.15.28:0/3132163674 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3132163674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 0x7ff1ec0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f1400640 1 --2- 172.21.15.28:0/3132163674 >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 0x7ff1ec15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1ebe00640 1 -- 172.21.15.28:0/3132163674 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534953624 0 0) 0x7ff1ec15bf00 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1ebe00640 1 -- 172.21.15.28:0/3132163674 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d0003530 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3132163674 >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 msgr2=0x7ff1ec15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3132163674 >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 0x7ff1ec15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3132163674 >> v1:172.21.15.28:6789/0 conn(0x7ff1ec0b34f0 legacy=0x7ff1ec0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3132163674 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0cddb0 con 0x7ff1ec0af060 2024-03-21T08:41:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f1400640 1 --2- 172.21.15.28:0/3132163674 >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 0x7ff1ec15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.553+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3132163674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 0x7ff1ec0cd380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097c0 tx=0x7ff1dc01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961d336655c0b4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1ebe00640 1 -- 172.21.15.28:0/3132163674 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0b0070 con 0x7ff1ec0af060 2024-03-21T08:41:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1ebe00640 1 -- 172.21.15.28:0/3132163674 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0ab600 con 0x7ff1ec0af060 2024-03-21T08:41:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1ebe00640 1 -- 172.21.15.28:0/3132163674 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0ab9d0 con 0x7ff1ec0af060 2024-03-21T08:41:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- 172.21.15.28:0/3132163674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 msgr2=0x7ff1ec0cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3132163674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 0x7ff1ec0cd380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097c0 tx=0x7ff1dc01c410 comp rx=0 tx=0).stop 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- 172.21.15.28:0/3132163674 shutdown_connections 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3132163674 >> v2:172.21.15.151:3300/0 conn(0x7ff1ec0b5520 0x7ff1ec15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3132163674 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0af060 0x7ff1ec0cd380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- 172.21.15.28:0/3132163674 >> 172.21.15.28:0/3132163674 conn(0x7ff1ec0cc230 msgr2=0x7ff1ec0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- 172.21.15.28:0/3132163674 shutdown_connections 2024-03-21T08:41:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- 172.21.15.28:0/3132163674 wait complete. 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 Processor -- start 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- start start 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 0x7ff1ec148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 0x7ff1ec0adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15d0d0 con 0x7ff1ec0b5520 2024-03-21T08:41:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15cf50 con 0x7ff1ec14f120 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f3400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15d250 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f0a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 0x7ff1ec148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35154/0 (socket says 172.21.15.28:35154) 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1f1400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 0x7ff1ec0adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.557+0000 7ff1e3e00640 1 -- 172.21.15.28:0/3803433561 learned_addr learned my addr 172.21.15.28:0/3803433561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 msgr2=0x7ff1ec0adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 0x7ff1ec0adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3803433561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 msgr2=0x7ff1ec14ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 -- 172.21.15.28:0/3803433561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec15b1f0 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e3e00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f1400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 0x7ff1ec0adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f0a00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 0x7ff1ec148ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ab000 tx=0x7ff1dc01ac50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0b0050 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0bc070 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc006e20 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec15b4d0 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec15df20 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec04f410 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff1dc0b5870 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 0x7ff1c8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.561+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff1dc120810 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.565+0000 7ff1e3e00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 0x7ff1c8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.565+0000 7ff1e3e00640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 0x7ff1c8069fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0aedd0 tx=0x7ff1d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.569+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff1dc016090 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.769+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1ec0728d0 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.769+0000 7ff1e9600640 1 -- 172.21.15.28:0/3803433561 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff1dc0f0e40 con 0x7ff1ec0b34f0 2024-03-21T08:41:40.774 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 msgr2=0x7ff1c8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 0x7ff1c8069fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0aedd0 tx=0x7ff1d4009040 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 msgr2=0x7ff1ec148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 0x7ff1ec148ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ab000 tx=0x7ff1dc01ac50 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 shutdown_connections 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1ec14f120 0x7ff1ec0adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec0b5520 0x7ff1ec14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff1c8067b20 0x7ff1c8069fe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 --2- 172.21.15.28:0/3803433561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff1ec0b34f0 0x7ff1ec148ea0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 >> 172.21.15.28:0/3803433561 conn(0x7ff1ec0cc230 msgr2=0x7ff1ec0b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 shutdown_connections 2024-03-21T08:41:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:40.773+0000 7ff1f3400640 1 -- 172.21.15.28:0/3803433561 wait complete. 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:41:40.794 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:41:40.795 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:41:40.795 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-21T08:41:40.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.795 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:41:40.804 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.804 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42369sec preferred_lft 42369sec 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:41:40.805 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.806 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:41:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-03-21T08:41:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:40.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.807 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:41:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:41:40.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.872 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:41:40.925 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:41:40.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:41:40.927 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:41.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.018 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:41:41.027 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:41:41.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.028 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-03-21T08:41:41.079 INFO:teuthology.orchestra.run.smithi028.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-21T08:41:41.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.080 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:41.080 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:41.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:41:41.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-21T08:41:41.080 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:41.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-21T08:41:41.223 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:41.223 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:41.223 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:41.223 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.223 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 6a:dc:9e:53:17:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::68dc:9eff:fe53:1748/64 scope link tentative 2024-03-21T08:41:41.224 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-21T08:41:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:41.287 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:41.287 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:41.287 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 6a:dc:9e:53:17:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::68dc:9eff:fe53:1748/64 scope link 2024-03-21T08:41:41.288 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:41.289 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-21T08:41:41.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-21T08:41:41.290 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:41.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.428 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:41:41.428 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:41:41.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.1 up 2024-03-21T08:41:41.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.1 master ceph-brx 2024-03-21T08:41:41.428 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:41:41.466 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-21T08:41:41.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:41.467 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-21T08:41:41.520 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:41.520 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:41.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:41.636 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:41.693 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:41.694 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:41:41.695 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:41:41.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.696 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:41.747 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:41:41.747 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-21T08:41:41.747 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-21T08:41:41.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:41:41.816 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:41:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:41:41.874 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:41:41.875 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:41:41.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:41.876 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:41:41.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:2024-03-21T08:41:41.885+0000 7fb42e046580 -1 init, newargv = 0x5581e8951a50 newargc=15 2024-03-21T08:41:41.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:ceph-fuse[34144]: starting ceph client 2024-03-21T08:41:41.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:ceph-fuse[34144]: starting fuse 2024-03-21T08:41:41.897 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:41:41.897 INFO:teuthology.orchestra.run.smithi028.stdout:44 2024-03-21T08:41:41.898 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-21T08:41:41.898 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:41:41.899 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:41:41.900 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.2")) 2024-03-21T08:41:41.900 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:41:42.026 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.2.34144.asok 2024-03-21T08:41:42.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:41:42.038 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.34144.asok 2024-03-21T08:41:42.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.34144.asok status 2024-03-21T08:41:42.211 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "2", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "34144", 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4875, 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:41:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4875 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.2:0", 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 391720632 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.2:0", 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 391720632 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:41:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4875 192.168.0.2:0/391720632", 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.2:0/391720632", 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 9, 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 25, 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:41:42.214 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:41:42.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:41:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 Processor -- start 2024-03-21T08:41:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 -- start start 2024-03-21T08:41:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 0x7f7ed815e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed815bed0 con 0x7f7ed815ba70 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed815ebf0 con 0x7f7ed80b4380 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7edd000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed815eed0 con 0x7f7ed80ce490 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ecd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ed815ba70 0x7f7ed80ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56894/0 (socket says 172.21.15.28:56894) 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ecd400640 1 -- 172.21.15.28:0/3657226699 learned_addr learned my addr 172.21.15.28:0/3657226699 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/3657226699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 0x7f7ed815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661678880 0 0) 0x7f7ed815bed0 con 0x7f7ed815ba70 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ebc003530 con 0x7f7ed815ba70 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150770691 0 0) 0x7f7ed815ebf0 con 0x7f7ed80b4380 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed815bed0 con 0x7f7ed80b4380 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5e00640 1 -- 172.21.15.28:0/3657226699 >> v1:172.21.15.151:6789/0 conn(0x7f7ed80b4380 legacy=0x7f7ed80b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5e00640 1 -- 172.21.15.28:0/3657226699 >> v1:172.21.15.28:6789/0 conn(0x7f7ed815ba70 legacy=0x7f7ed80ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.365+0000 7f7ed5e00640 1 -- 172.21.15.28:0/3657226699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed815f1b0 con 0x7f7ed80ce490 2024-03-21T08:41:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/3657226699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 0x7f7ed815e610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80097f0 tx=0x7f7ec801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ffc86dd784042a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80b0070 con 0x7f7ed80ce490 2024-03-21T08:41:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec80ab600 con 0x7f7ed80ce490 2024-03-21T08:41:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7ed5400640 1 -- 172.21.15.28:0/3657226699 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80ab9d0 con 0x7f7ed80ce490 2024-03-21T08:41:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 -- 172.21.15.28:0/3657226699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 msgr2=0x7f7ed815e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 --2- 172.21.15.28:0/3657226699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 0x7f7ed815e610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80097f0 tx=0x7f7ec801c450 comp rx=0 tx=0).stop 2024-03-21T08:41:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 -- 172.21.15.28:0/3657226699 shutdown_connections 2024-03-21T08:41:42.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 --2- 172.21.15.28:0/3657226699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80ce490 0x7f7ed815e610 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 -- 172.21.15.28:0/3657226699 >> 172.21.15.28:0/3657226699 conn(0x7f7ed80cc230 msgr2=0x7f7ed80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:42.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 -- 172.21.15.28:0/3657226699 shutdown_connections 2024-03-21T08:41:42.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.369+0000 7f7edd000640 1 -- 172.21.15.28:0/3657226699 wait complete. 2024-03-21T08:41:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 Processor -- start 2024-03-21T08:41:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- start start 2024-03-21T08:41:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 0x7f7ed814eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ed815ba70 0x7f7ed814eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed8168b40 con 0x7f7ed80ce490 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed81689c0 con 0x7f7ed815ba70 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed8168cc0 con 0x7f7ed80b4380 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 0x7f7ed814eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed6800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ed815ba70 0x7f7ed814eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 0x7f7ed814eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35168/0 (socket says 172.21.15.28:35168) 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecd400640 1 -- 172.21.15.28:0/4089008521 learned_addr learned my addr 172.21.15.28:0/4089008521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:53854/0 (socket says 172.21.15.28:53854) 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 -- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ed815ba70 msgr2=0x7f7ed814eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ed815ba70 0x7f7ed814eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 -- 172.21.15.28:0/4089008521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 msgr2=0x7f7ed814eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 0x7f7ed814eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 -- 172.21.15.28:0/4089008521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed806eb30 con 0x7f7ed80b4380 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ed5e00640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80731c0 tx=0x7f7ec80191f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80b0040 con 0x7f7ed80b4380 2024-03-21T08:41:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec80ab400 con 0x7f7ed80b4380 2024-03-21T08:41:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8006e20 con 0x7f7ed80b4380 2024-03-21T08:41:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed806ee10 con 0x7f7ed80b4380 2024-03-21T08:41:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed806f3d0 con 0x7f7ed80b4380 2024-03-21T08:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.373+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed804f410 con 0x7f7ed80b4380 2024-03-21T08:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.377+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7ec80bd030 con 0x7f7ed80b4380 2024-03-21T08:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.377+0000 7f7ecea00640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 0x7f7eb406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.377+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7ec811f920 con 0x7f7ed80b4380 2024-03-21T08:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.377+0000 7f7ecd400640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 0x7f7eb406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:42.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.377+0000 7f7ecd400640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 0x7f7eb406a010 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ed815c770 tx=0x7f7ec0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:42.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.381+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ec80c4020 con 0x7f7ed80b4380 2024-03-21T08:41:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.589+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7ed8072c10 con 0x7f7ed80b4380 2024-03-21T08:41:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.589+0000 7f7ecea00640 1 -- 172.21.15.28:0/4089008521 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7ec80160d0 con 0x7f7ed80b4380 2024-03-21T08:41:42.594 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 msgr2=0x7f7eb406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 0x7f7eb406a010 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ed815c770 tx=0x7f7ec0009040 comp rx=0 tx=0).stop 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 msgr2=0x7f7ed815d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80731c0 tx=0x7f7ec80191f0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 shutdown_connections 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ed815ba70 0x7f7ed814eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed80ce490 0x7f7ed814eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7eb4067b50 0x7f7eb406a010 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 --2- 172.21.15.28:0/4089008521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ed80b4380 0x7f7ed815d2c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 >> 172.21.15.28:0/4089008521 conn(0x7f7ed80cc230 msgr2=0x7f7ed80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 shutdown_connections 2024-03-21T08:41:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:42.593+0000 7f7edd000640 1 -- 172.21.15.28:0/4089008521 wait complete. 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi151.front.sepia.ceph.com 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:41:42.615 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:41:42.616 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-21T08:41:42.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.616 DEBUG:teuthology.orchestra.run.smithi151:> ip addr 2024-03-21T08:41:42.625 INFO:teuthology.orchestra.run.smithi151.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f0 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f1 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:62 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:63 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: inet 172.21.15.151/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft 42451sec preferred_lft 42451sec 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:42.627 INFO:teuthology.orchestra.run.smithi151.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:41:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 02:42:72:1e:5d:fe brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:41:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:42.628 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-21T08:41:42.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.628 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link add name ceph-brx type bridge 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip addr flush dev ceph-brx 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set ceph-brx up 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-21T08:41:42.629 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:42.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.740 DEBUG:teuthology.orchestra.run.smithi151:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:41:42.757 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-21T08:41:42.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.760 DEBUG:teuthology.orchestra.run.smithi151:> route 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:Kernel IP routing table 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:42.861 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:42.862 INFO:teuthology.orchestra.run.smithi151.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:42.862 INFO:teuthology.orchestra.run.smithi151.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:41:42.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-21T08:41:42.862 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:42.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.912 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-03-21T08:41:42.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.921 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list-id 2024-03-21T08:41:42.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:42.970 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:42.970 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:42.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:42.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-21T08:41:42.971 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:43.052 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-21T08:41:43.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-21T08:41:43.053 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:43.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:43.230 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:43.230 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:43.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set brx.0 up 2024-03-21T08:41:43.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set dev brx.0 master ceph-brx 2024-03-21T08:41:43.230 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:43.271 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-21T08:41:43.271 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:43.271 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:43.322 INFO:teuthology.orchestra.run.smithi151.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-21T08:41:43.322 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:43.322 DEBUG:teuthology.orchestra.run.smithi151:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:43.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-03-21T08:41:43.434 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/run/lock 2024-03-21T08:41:43.487 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/fs/nfsd 2024-03-21T08:41:43.488 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/run/rpc_pipefs 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-4 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-5 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-6 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-7 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-03-21T08:41:43.489 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:43.490 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:43.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:43.491 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-03-21T08:41:43.546 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:41:43.546 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-21T08:41:43.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-03-21T08:41:43.614 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-03-21T08:41:43.663 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/run/lock 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/fs/nfsd 2024-03-21T08:41:43.664 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/rpc_pipefs 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-4 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-5 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-6 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-7 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:43.665 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:43.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:43.666 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-03-21T08:41:43.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi151.stderr:2024-03-21T08:41:43.700+0000 7fbf455fc580 -1 init, newargv = 0x55b5378b7b50 newargc=15 2024-03-21T08:41:43.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi151.stderr:ceph-fuse[32726]: starting ceph client 2024-03-21T08:41:43.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi151.stderr:ceph-fuse[32726]: starting fuse 2024-03-21T08:41:43.716 INFO:teuthology.orchestra.run.smithi151.stdout:43 2024-03-21T08:41:43.716 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:41:43.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:41:43.716 DEBUG:teuthology.orchestra.run.smithi151:> import glob 2024-03-21T08:41:43.716 DEBUG:teuthology.orchestra.run.smithi151:> import re 2024-03-21T08:41:43.716 DEBUG:teuthology.orchestra.run.smithi151:> import os 2024-03-21T08:41:43.716 DEBUG:teuthology.orchestra.run.smithi151:> import subprocess 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> def _find_admin_socket(client_name): 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> files = glob.glob(asok_path) 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> # Given a non-glob path, it better be there 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> if "*" not in asok_path: 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> assert(len(files) == 1) 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> return files[0] 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> for f in files: 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:41:43.717 DEBUG:teuthology.orchestra.run.smithi151:> contents = proc_f.read() 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> if mountpoint in contents: 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> return f 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> print(_find_admin_socket("client.1")) 2024-03-21T08:41:43.718 DEBUG:teuthology.orchestra.run.smithi151:> ' 2024-03-21T08:41:43.838 INFO:teuthology.orchestra.run.smithi151.stdout:/var/run/ceph/ceph-client.1.32726.asok 2024-03-21T08:41:43.842 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:41:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:41:43.850 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.32726.asok 2024-03-21T08:41:43.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:43.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.32726.asok status 2024-03-21T08:41:44.021 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-21T08:41:44.021 INFO:teuthology.orchestra.run.smithi151.stdout: "metadata": { 2024-03-21T08:41:44.021 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:41:44.021 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "entity_id": "1", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "hostname": "smithi151", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "pid": "32726", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "root": "/" 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_count": 0, 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_pinned_count": 0, 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "id": 4873, 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "inst": { 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "name": { 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "client", 2024-03-21T08:41:44.022 INFO:teuthology.orchestra.run.smithi151.stdout: "num": 4873 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 4065465614 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 4065465614 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "inst_str": "client.4873 192.168.0.1:0/4065465614", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr_str": "192.168.0.1:0/4065465614", 2024-03-21T08:41:44.023 INFO:teuthology.orchestra.run.smithi151.stdout: "inode_count": 1, 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_epoch": 9, 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch": 25, 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklisted": false, 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout: "fs_name": "cephfs" 2024-03-21T08:41:44.024 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-21T08:41:44.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:41:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 Processor -- start 2024-03-21T08:41:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 -- start start 2024-03-21T08:41:44.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 0x7ff1900cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1900cd5d0 0x7ff190099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900cdb50 con 0x7ff19009a0c0 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900c5530 con 0x7ff1900cd5d0 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff197400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900cde30 con 0x7ff19015f260 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1900cd5d0 0x7ff190099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff194a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 0x7ff1900cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff195400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7ff19009a0c0 0x7ff1900cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:34754/0 (socket says 172.21.15.28:34754) 2024-03-21T08:41:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff195400640 1 -- 172.21.15.28:0/3192975031 learned_addr learned my addr 172.21.15.28:0/3192975031 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18f400640 1 -- 172.21.15.28:0/3192975031 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334350803 0 0) 0x7ff1900cdb50 con 0x7ff19009a0c0 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18f400640 1 -- 172.21.15.28:0/3192975031 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff178003530 con 0x7ff19009a0c0 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18fe00640 1 -- 172.21.15.28:0/3192975031 >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 msgr2=0x7ff1900cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18fe00640 1 --2- 172.21.15.28:0/3192975031 >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 0x7ff1900cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18fe00640 1 -- 172.21.15.28:0/3192975031 >> v1:172.21.15.151:6789/0 conn(0x7ff19009a0c0 legacy=0x7ff1900cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff18fe00640 1 -- 172.21.15.28:0/3192975031 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900c5780 con 0x7ff1900cd5d0 2024-03-21T08:41:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.189+0000 7ff194a00640 1 --2- 172.21.15.28:0/3192975031 >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 0x7ff1900cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff18fe00640 1 --2- 172.21.15.28:0/3192975031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900cd5d0 0x7ff190099b80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff1900b4110 tx=0x7ff17c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3015fa4725997db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff18f400640 1 -- 172.21.15.28:0/3192975031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0b0070 con 0x7ff1900cd5d0 2024-03-21T08:41:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff18f400640 1 -- 172.21.15.28:0/3192975031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c0ab600 con 0x7ff1900cd5d0 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff18f400640 1 -- 172.21.15.28:0/3192975031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0ab9d0 con 0x7ff1900cd5d0 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 -- 172.21.15.28:0/3192975031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900cd5d0 msgr2=0x7ff190099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 --2- 172.21.15.28:0/3192975031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900cd5d0 0x7ff190099b80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff1900b4110 tx=0x7ff17c01c3f0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 -- 172.21.15.28:0/3192975031 shutdown_connections 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 --2- 172.21.15.28:0/3192975031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900cd5d0 0x7ff190099b80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 --2- 172.21.15.28:0/3192975031 >> v2:172.21.15.151:3300/0 conn(0x7ff19015f260 0x7ff1900cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 -- 172.21.15.28:0/3192975031 >> 172.21.15.28:0/3192975031 conn(0x7ff1900b34d0 msgr2=0x7ff1900b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 -- 172.21.15.28:0/3192975031 shutdown_connections 2024-03-21T08:41:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.193+0000 7ff197400640 1 -- 172.21.15.28:0/3192975031 wait complete. 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 Processor -- start 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- start start 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 0x7ff1900caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 0x7ff19014da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 0x7ff19014df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190168bc0 con 0x7ff19015f260 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190168a40 con 0x7ff1900cd5d0 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff190168d40 con 0x7ff19009a0c0 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 0x7ff19014df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff194a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 0x7ff1900caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff18fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 0x7ff19014da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff194a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 0x7ff1900caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:53856/0 (socket says 172.21.15.28:53856) 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff194a00640 1 -- 172.21.15.28:0/4278189746 learned_addr learned my addr 172.21.15.28:0/4278189746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 -- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 msgr2=0x7ff1900caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 0x7ff1900caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 -- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 msgr2=0x7ff19014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 0x7ff19014da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 -- 172.21.15.28:0/4278189746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900c9900 con 0x7ff19015f260 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff18fe00640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 0x7ff19014da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff195400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 0x7ff19014df50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff18800cad0 tx=0x7ff18800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188017070 con 0x7ff19015f260 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18800ab70 con 0x7ff19015f260 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188012400 con 0x7ff19015f260 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1900c9be0 con 0x7ff19015f260 2024-03-21T08:41:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1900ca1a0 con 0x7ff19015f260 2024-03-21T08:41:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.197+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19004f410 con 0x7ff19015f260 2024-03-21T08:41:44.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.201+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff18801a070 con 0x7ff19015f260 2024-03-21T08:41:44.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.201+0000 7ff18cc00640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 0x7ff170069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:44.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.201+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff188085e20 con 0x7ff19015f260 2024-03-21T08:41:44.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.201+0000 7ff194a00640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 0x7ff170069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:44.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.201+0000 7ff194a00640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 0x7ff170069fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff1901fa340 tx=0x7ff184009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.205+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff188055ea0 con 0x7ff19015f260 2024-03-21T08:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.405+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff190163c60 con 0x7ff19015f260 2024-03-21T08:41:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.405+0000 7ff18cc00640 1 -- 172.21.15.28:0/4278189746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff188059f40 con 0x7ff19015f260 2024-03-21T08:41:44.410 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-21T08:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 msgr2=0x7ff170069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 0x7ff170069fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff1901fa340 tx=0x7ff184009190 comp rx=0 tx=0).stop 2024-03-21T08:41:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 msgr2=0x7ff19014df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 0x7ff19014df50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff18800cad0 tx=0x7ff18800ced0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 shutdown_connections 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015f260 0x7ff19014df50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff1900cd5d0 0x7ff19014da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff170067b20 0x7ff170069fe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 --2- 172.21.15.28:0/4278189746 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff19009a0c0 0x7ff1900caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 >> 172.21.15.28:0/4278189746 conn(0x7ff1900b34d0 msgr2=0x7ff19015dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 shutdown_connections 2024-03-21T08:41:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:44.409+0000 7ff197400640 1 -- 172.21.15.28:0/4278189746 wait complete. 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi151.front.sepia.ceph.com 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:41:44.432 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-21T08:41:44.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.433 DEBUG:teuthology.orchestra.run.smithi151:> ip addr 2024-03-21T08:41:44.441 INFO:teuthology.orchestra.run.smithi151.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:44.441 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f0 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:da:6e:0d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: altname enp7s0f1 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:41:44.442 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:62 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0c:c4:7a:88:6f:63 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: inet 172.21.15.151/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft 42449sec preferred_lft 42449sec 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:41:44.443 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 02:42:72:1e:5d:fe brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-03-21T08:41:44.444 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.445 DEBUG:teuthology.orchestra.run.smithi151:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:41:44.503 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-21T08:41:44.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.505 DEBUG:teuthology.orchestra.run.smithi151:> route 2024-03-21T08:41:44.557 INFO:teuthology.orchestra.run.smithi151.stdout:Kernel IP routing table 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-21T08:41:44.558 INFO:teuthology.orchestra.run.smithi151.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:41:44.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-21T08:41:44.559 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:44.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.647 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-03-21T08:41:44.655 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-21T08:41:44.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.656 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list-id 2024-03-21T08:41:44.705 INFO:teuthology.orchestra.run.smithi151.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-21T08:41:44.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.707 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:44.707 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:44.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:41:44.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-21T08:41:44.708 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:44.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-21T08:41:44.853 INFO:teuthology.orchestra.run.smithi151.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:44.853 INFO:teuthology.orchestra.run.smithi151.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether a2:f2:1a:0c:e8:74 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-21T08:41:44.854 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.855 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::a0f2:1aff:fe0c:e874/64 scope link 2024-03-21T08:41:44.855 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 ::1/128 scope host 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: link/ether a2:f2:1a:0c:e8:74 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: inet6 fe80::a0f2:1aff:fe0c:e874/64 scope link 2024-03-21T08:41:44.918 INFO:teuthology.orchestra.run.smithi151.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:41:44.920 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-21T08:41:44.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-21T08:41:44.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-21T08:41:44.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-21T08:41:44.921 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:45.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.061 DEBUG:teuthology.orchestra.run.smithi151:> (cd / && exec bash -c ' 2024-03-21T08:41:45.061 DEBUG:teuthology.orchestra.run.smithi151:> set -e 2024-03-21T08:41:45.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set brx.1 up 2024-03-21T08:41:45.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link set dev brx.1 master ceph-brx 2024-03-21T08:41:45.061 DEBUG:teuthology.orchestra.run.smithi151:> ') 2024-03-21T08:41:45.100 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-21T08:41:45.100 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:45.100 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:45.155 INFO:teuthology.orchestra.run.smithi151.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-21T08:41:45.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:41:45.156 DEBUG:teuthology.orchestra.run.smithi151:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:45.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-03-21T08:41:45.271 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/run/lock 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-03-21T08:41:45.324 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/fs/nfsd 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/run/rpc_pipefs 2024-03-21T08:41:45.325 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-4 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-5 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-6 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-7 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:45.326 INFO:teuthology.orchestra.run.smithi151.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:45.327 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:41:45.327 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:41:45.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.328 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-03-21T08:41:45.377 INFO:teuthology.orchestra.run.smithi151.stdout:43 2024-03-21T08:41:45.378 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-21T08:41:45.378 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-21T08:41:45.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo modprobe fuse 2024-03-21T08:41:45.445 DEBUG:teuthology.orchestra.run.smithi151:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:41:45.495 INFO:teuthology.orchestra.run.smithi151.stdout:/sys 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/proc 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/dev 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/pts 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/run 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/ 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/security 2024-03-21T08:41:45.496 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/shm 2024-03-21T08:41:45.497 INFO:teuthology.orchestra.run.smithi151.stdout:/run/lock 2024-03-21T08:41:45.497 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/cgroup 2024-03-21T08:41:45.497 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/pstore 2024-03-21T08:41:45.497 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/bpf 2024-03-21T08:41:45.497 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:41:45.498 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/hugepages 2024-03-21T08:41:45.498 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/mqueue 2024-03-21T08:41:45.498 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/debug 2024-03-21T08:41:45.498 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/tracing 2024-03-21T08:41:45.498 INFO:teuthology.orchestra.run.smithi151.stdout:/proc/fs/nfsd 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/fs/fuse/connections 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/sys/kernel/config 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/run/rpc_pipefs 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/run/user/1000 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-4 2024-03-21T08:41:45.499 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-5 2024-03-21T08:41:45.500 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-6 2024-03-21T08:41:45.500 INFO:teuthology.orchestra.run.smithi151.stdout:/var/lib/ceph/osd/ceph-7 2024-03-21T08:41:45.500 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns 2024-03-21T08:41:45.500 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:45.501 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:41:45.501 INFO:teuthology.orchestra.run.smithi151.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:45.502 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:41:45.502 INFO:teuthology.orchestra.run.smithi151.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:41:45.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.505 DEBUG:teuthology.orchestra.run.smithi151:> ls /sys/fs/fuse/connections 2024-03-21T08:41:45.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi151.stderr:2024-03-21T08:41:45.516+0000 7fd3b782c580 -1 init, newargv = 0x5591455be520 newargc=15 2024-03-21T08:41:45.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi151.stderr:ceph-fuse[32832]: starting ceph client 2024-03-21T08:41:45.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi151.stderr:ceph-fuse[32832]: starting fuse 2024-03-21T08:41:45.532 INFO:teuthology.orchestra.run.smithi151.stdout:43 2024-03-21T08:41:45.532 INFO:teuthology.orchestra.run.smithi151.stdout:44 2024-03-21T08:41:45.532 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-21T08:41:45.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:41:45.532 DEBUG:teuthology.orchestra.run.smithi151:> import glob 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> import re 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> import os 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> import subprocess 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> def _find_admin_socket(client_name): 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> files = glob.glob(asok_path) 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> # Given a non-glob path, it better be there 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> if "*" not in asok_path: 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> assert(len(files) == 1) 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> return files[0] 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> for f in files: 2024-03-21T08:41:45.533 DEBUG:teuthology.orchestra.run.smithi151:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> contents = proc_f.read() 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> if mountpoint in contents: 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> return f 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> print(_find_admin_socket("client.3")) 2024-03-21T08:41:45.534 DEBUG:teuthology.orchestra.run.smithi151:> ' 2024-03-21T08:41:45.671 INFO:teuthology.orchestra.run.smithi151.stdout:/var/run/ceph/ceph-client.3.32832.asok 2024-03-21T08:41:45.676 INFO:teuthology.orchestra.run.smithi151.stderr:2024-03-21T08:41:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:41:45.684 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.32832.asok 2024-03-21T08:41:45.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.32832.asok status 2024-03-21T08:41:45.854 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-21T08:41:45.854 INFO:teuthology.orchestra.run.smithi151.stdout: "metadata": { 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "entity_id": "3", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "hostname": "smithi151", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "pid": "32832", 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "root": "/" 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_count": 0, 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "dentry_pinned_count": 0, 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "id": 4888, 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "inst": { 2024-03-21T08:41:45.855 INFO:teuthology.orchestra.run.smithi151.stdout: "name": { 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "client", 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "num": 4888 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.2:0", 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 156533072 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": { 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "192.168.0.2:0", 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 156533072 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-03-21T08:41:45.856 INFO:teuthology.orchestra.run.smithi151.stdout: "inst_str": "client.4888 192.168.0.2:0/156533072", 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "addr_str": "192.168.0.2:0/156533072", 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "inode_count": 1, 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_epoch": 9, 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch": 25, 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklisted": false, 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout: "fs_name": "cephfs" 2024-03-21T08:41:45.857 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-21T08:41:45.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.869 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:41:45.869 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:45.880 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:41:45.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:45.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:41:45.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.946 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:41:45.946 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:45.997 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:41:45.998 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:45.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:45.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-21T08:41:46.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:46.066 DEBUG:teuthology.orchestra.run.smithi151:> stat --file-system '--printf=%T 2024-03-21T08:41:46.066 DEBUG:teuthology.orchestra.run.smithi151:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:46.075 INFO:teuthology.orchestra.run.smithi151.stdout:fuseblk 2024-03-21T08:41:46.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:46.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:46.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-21T08:41:46.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:46.138 DEBUG:teuthology.orchestra.run.smithi151:> stat --file-system '--printf=%T 2024-03-21T08:41:46.138 DEBUG:teuthology.orchestra.run.smithi151:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:46.191 INFO:teuthology.orchestra.run.smithi151.stdout:fuseblk 2024-03-21T08:41:46.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:46.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:41:46.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-21T08:41:46.258 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-21T08:41:46.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:46.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 Processor -- start 2024-03-21T08:41:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 -- start start 2024-03-21T08:41:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbd98099b80 0x7fbd98099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd980cdb50 con 0x7fbd9815f260 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9809a520 con 0x7fbd98099b80 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbda0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980c5530 con 0x7fbd980c5f80 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbd98099b80 0x7fbd98099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbd98099b80 0x7fbd98099fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35196/0 (socket says 172.21.15.28:35196) 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 -- 172.21.15.28:0/1379329870 learned_addr learned my addr 172.21.15.28:0/1379329870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9e000640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 -- 172.21.15.28:0/1379329870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 msgr2=0x7fbd980c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 -- 172.21.15.28:0/1379329870 >> v1:172.21.15.151:6789/0 conn(0x7fbd9815f260 legacy=0x7fbd980cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd9cc00640 1 -- 172.21.15.28:0/1379329870 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980cde30 con 0x7fbd98099b80 2024-03-21T08:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.425+0000 7fbd97e00640 1 -- 172.21.15.28:0/1379329870 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027990532 0 0) 0x7fbd980cdb50 con 0x7fbd9815f260 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbd9cc00640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98099fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009870 tx=0x7fbd8401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3d55d9d6aac541 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbd97e00640 1 -- 172.21.15.28:0/1379329870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840aa600 con 0x7fbd98099b80 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbd97e00640 1 -- 172.21.15.28:0/1379329870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840af070 con 0x7fbd98099b80 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbd97e00640 1 -- 172.21.15.28:0/1379329870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840aa9c0 con 0x7fbd98099b80 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 -- 172.21.15.28:0/1379329870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 msgr2=0x7fbd98099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98099fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009870 tx=0x7fbd8401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 -- 172.21.15.28:0/1379329870 shutdown_connections 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 --2- 172.21.15.28:0/1379329870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98099fe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 -- 172.21.15.28:0/1379329870 >> 172.21.15.28:0/1379329870 conn(0x7fbd980b34d0 msgr2=0x7fbd980b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 -- 172.21.15.28:0/1379329870 shutdown_connections 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.429+0000 7fbda0000640 1 -- 172.21.15.28:0/1379329870 wait complete. 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 Processor -- start 2024-03-21T08:41:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- start start 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 0x7fbd9815de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980cd7a0 con 0x7fbd98099b80 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9815f640 con 0x7fbd9815f260 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98164ef0 con 0x7fbd980c5f80 2024-03-21T08:41:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35206/0 (socket says 172.21.15.28:35206) 2024-03-21T08:41:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 0x7fbd9815de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9d600640 1 -- 172.21.15.28:0/2633229830 learned_addr learned my addr 172.21.15.28:0/2633229830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 -- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 msgr2=0x7fbd9815de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 0x7fbd9815de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 -- 172.21.15.28:0/2633229830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 msgr2=0x7fbd98147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 -- 172.21.15.28:0/2633229830 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd981651d0 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9e000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 0x7fbd9815de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9d600640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd9cc00640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980b47c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd840ab000 tx=0x7fbd8401ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840bd5f0 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840bd7b0 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840bdba0 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd981654b0 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd98147850 con 0x7fbd980c5f80 2024-03-21T08:41:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.433+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9804f410 con 0x7fbd980c5f80 2024-03-21T08:41:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.437+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbd840a9070 con 0x7fbd980c5f80 2024-03-21T08:41:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.437+0000 7fbd95600640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 0x7fbd78069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.437+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbd84120840 con 0x7fbd980c5f80 2024-03-21T08:41:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.437+0000 7fbd9d600640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 0x7fbd78069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.437+0000 7fbd9d600640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 0x7fbd78069fe0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c007fd0 tx=0x7fbd8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.441+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbd84016150 con 0x7fbd980c5f80 2024-03-21T08:41:46.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.605+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd98074180 con 0x7fbd980c5f80 2024-03-21T08:41:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbd95600640 1 -- 172.21.15.28:0/2633229830 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18526 (secure 0 0 0) 0x7fbd840f0830 con 0x7fbd980c5f80 2024-03-21T08:41:46.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:46.613 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:12.439947+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:46.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 msgr2=0x7fbd78069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 0x7fbd78069fe0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c007fd0 tx=0x7fbd8c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:46.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 msgr2=0x7fbd980b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980b47c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd840ab000 tx=0x7fbd8401ace0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 shutdown_connections 2024-03-21T08:41:46.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd9815f260 0x7fbd9815de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd78067b20 0x7fbd78069fe0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd980c5f80 0x7fbd980b47c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 --2- 172.21.15.28:0/2633229830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd98099b80 0x7fbd98147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 >> 172.21.15.28:0/2633229830 conn(0x7fbd980b34d0 msgr2=0x7fbd98073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:46.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 shutdown_connections 2024-03-21T08:41:46.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.613+0000 7fbda0000640 1 -- 172.21.15.28:0/2633229830 wait complete. 2024-03-21T08:41:46.634 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:46.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 Processor -- start 2024-03-21T08:41:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 -- start start 2024-03-21T08:41:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef140cde40 0x7fef140cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1406f500 con 0x7fef1406f0a0 2024-03-21T08:41:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef140ce5b0 con 0x7fef14071a80 2024-03-21T08:41:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef1aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1415c720 con 0x7fef140cde40 2024-03-21T08:41:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef18a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef140cde40 0x7fef140cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef13e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fef14071a80 0x7fef14071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:34772/0 (socket says 172.21.15.28:34772) 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef13e00640 1 -- 172.21.15.28:0/539435730 learned_addr learned my addr 172.21.15.28:0/539435730 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.773+0000 7fef0b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fef1406f0a0 0x7fef1415cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56910/0 (socket says 172.21.15.28:56910) 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef18a00640 1 -- 172.21.15.28:0/539435730 >> v1:172.21.15.151:6789/0 conn(0x7fef14071a80 legacy=0x7fef14071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef18a00640 1 -- 172.21.15.28:0/539435730 >> v1:172.21.15.28:6789/0 conn(0x7fef1406f0a0 legacy=0x7fef1415cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef13400640 1 -- 172.21.15.28:0/539435730 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367814550 0 0) 0x7fef140ce5b0 con 0x7fef14071a80 2024-03-21T08:41:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef18a00640 1 -- 172.21.15.28:0/539435730 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef140ce890 con 0x7fef140cde40 2024-03-21T08:41:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef18a00640 1 --2- 172.21.15.28:0/539435730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef140cde40 0x7fef140cc4b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009d90 tx=0x7fef0c01c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be8136573033c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef13400640 1 -- 172.21.15.28:0/539435730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0a66d0 con 0x7fef140cde40 2024-03-21T08:41:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef13400640 1 -- 172.21.15.28:0/539435730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c0a6870 con 0x7fef140cde40 2024-03-21T08:41:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef13400640 1 -- 172.21.15.28:0/539435730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0a6c40 con 0x7fef140cde40 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef1aa00640 1 -- 172.21.15.28:0/539435730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef140cde40 msgr2=0x7fef140cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.777+0000 7fef1aa00640 1 --2- 172.21.15.28:0/539435730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef140cde40 0x7fef140cc4b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009d90 tx=0x7fef0c01c770 comp rx=0 tx=0).stop 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- 172.21.15.28:0/539435730 shutdown_connections 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 --2- 172.21.15.28:0/539435730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef140cde40 0x7fef140cc4b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- 172.21.15.28:0/539435730 >> 172.21.15.28:0/539435730 conn(0x7fef140674f0 msgr2=0x7fef1406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- 172.21.15.28:0/539435730 shutdown_connections 2024-03-21T08:41:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- 172.21.15.28:0/539435730 wait complete. 2024-03-21T08:41:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 Processor -- start 2024-03-21T08:41:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- start start 2024-03-21T08:41:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 0x7fef141f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 0x7fef14159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef14144b20 con 0x7fef1406f0a0 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef140cce00 con 0x7fef140cde40 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef14169f30 con 0x7fef14071a80 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35228/0 (socket says 172.21.15.28:35228) 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 -- 172.21.15.28:0/30300427 learned_addr learned my addr 172.21.15.28:0/30300427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef18a00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 0x7fef14159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef0b400640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 0x7fef141f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 -- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 msgr2=0x7fef141f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 0x7fef141f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 -- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 msgr2=0x7fef14159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 0x7fef14159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 -- 172.21.15.28:0/30300427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef14159580 con 0x7fef1406f0a0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef0b400640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 0x7fef141f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef18a00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 0x7fef14159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef13e00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c4a0 tx=0x7fef0400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04018070 con 0x7fef1406f0a0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef040074e0 con 0x7fef1406f0a0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04004a30 con 0x7fef1406f0a0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.781+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef14159820 con 0x7fef1406f0a0 2024-03-21T08:41:46.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef14157220 con 0x7fef1406f0a0 2024-03-21T08:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef14073430 con 0x7fef1406f0a0 2024-03-21T08:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fef04004bd0 con 0x7fef1406f0a0 2024-03-21T08:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef10c00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 0x7feef0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:46.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fef04085620 con 0x7fef1406f0a0 2024-03-21T08:41:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef0b400640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 0x7feef0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.785+0000 7fef0b400640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 0x7feef0069fe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef1413e160 tx=0x7feefc007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:46.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.789+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fef04055cd0 con 0x7fef1406f0a0 2024-03-21T08:41:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef1404f4f0 con 0x7fef1406f0a0 2024-03-21T08:41:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef10c00640 1 -- 172.21.15.28:0/30300427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fef040555d0 con 0x7fef1406f0a0 2024-03-21T08:41:46.957 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:46.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 msgr2=0x7feef0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 0x7feef0069fe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef1413e160 tx=0x7feefc007cf0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 msgr2=0x7fef1413af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c4a0 tx=0x7fef0400c960 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 shutdown_connections 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef140cde40 0x7fef14159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feef0067b20 0x7feef0069fe0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef14071a80 0x7fef141f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 --2- 172.21.15.28:0/30300427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef1406f0a0 0x7fef1413af50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 >> 172.21.15.28:0/30300427 conn(0x7fef140674f0 msgr2=0x7fef14071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 shutdown_connections 2024-03-21T08:41:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:46.953+0000 7fef1aa00640 1 -- 172.21.15.28:0/30300427 wait complete. 2024-03-21T08:41:46.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.117+0000 7f1429a00640 1 Processor -- start 2024-03-21T08:41:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.117+0000 7f1429a00640 1 -- start start 2024-03-21T08:41:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1429a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1424071a80 0x7f142406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1429a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f14240cde40 0x7f1424071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1429a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14240723a0 con 0x7f142415cbb0 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1429a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142415c720 con 0x7f14240cde40 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1429a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14240ce5b0 con 0x7f1424071a80 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1423400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f142415cbb0 0x7f14240cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56920/0 (socket says 172.21.15.28:56920) 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1423400640 1 -- 172.21.15.28:0/1965221070 learned_addr learned my addr 172.21.15.28:0/1965221070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f141be00640 1 --2- 172.21.15.28:0/1965221070 >> v2:172.21.15.151:3300/0 conn(0x7f14240cde40 0x7f1424071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1422a00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1424071a80 0x7f142406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1422000640 1 -- 172.21.15.28:0/1965221070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568702727 0 0) 0x7f14240723a0 con 0x7f142415cbb0 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f1422000640 1 -- 172.21.15.28:0/1965221070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408003530 con 0x7f142415cbb0 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f141be00640 1 -- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1424071a80 msgr2=0x7f142406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f141be00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1424071a80 0x7f142406f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f141be00640 1 -- 172.21.15.28:0/1965221070 >> v1:172.21.15.28:6789/0 conn(0x7f142415cbb0 legacy=0x7f14240cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.121+0000 7f141be00640 1 -- 172.21.15.28:0/1965221070 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240ce890 con 0x7f14240cde40 2024-03-21T08:41:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f141be00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424071e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f140c0097c0 tx=0x7f140c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1bfc9bfa5b0a715 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1422000640 1 -- 172.21.15.28:0/1965221070 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c0ab600 con 0x7f14240cde40 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1422000640 1 -- 172.21.15.28:0/1965221070 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c0b0070 con 0x7f14240cde40 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1422000640 1 -- 172.21.15.28:0/1965221070 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c0ab9a0 con 0x7f14240cde40 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 msgr2=0x7f1424071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424071e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f140c0097c0 tx=0x7f140c01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- 172.21.15.28:0/1965221070 shutdown_connections 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424071e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 --2- 172.21.15.28:0/1965221070 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1424071a80 0x7f142406f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- 172.21.15.28:0/1965221070 >> 172.21.15.28:0/1965221070 conn(0x7f14240674f0 msgr2=0x7f142406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- 172.21.15.28:0/1965221070 shutdown_connections 2024-03-21T08:41:47.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- 172.21.15.28:0/1965221070 wait complete. 2024-03-21T08:41:47.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 Processor -- start 2024-03-21T08:41:47.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.125+0000 7f1429a00640 1 -- start start 2024-03-21T08:41:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f142415cbb0 0x7f14240b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424168f20 con 0x7f1424071a80 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1424144b20 con 0x7f14240cde40 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14241690a0 con 0x7f142415cbb0 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35238/0 (socket says 172.21.15.28:35238) 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 -- 172.21.15.28:0/4100289114 learned_addr learned my addr 172.21.15.28:0/4100289114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1423400640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f142415cbb0 0x7f14240b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f141be00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 -- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f142415cbb0 msgr2=0x7f14240b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f142415cbb0 0x7f14240b1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 -- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 msgr2=0x7f1424157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 -- 172.21.15.28:0/4100289114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240b1da0 con 0x7f1424071a80 2024-03-21T08:41:47.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1422a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f141400e930 tx=0x7f141400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414017070 con 0x7f1424071a80 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400c9a0 con 0x7f1424071a80 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400cd50 con 0x7f1424071a80 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14240b3f90 con 0x7f1424071a80 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.129+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14240b4550 con 0x7f1424071a80 2024-03-21T08:41:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1424073430 con 0x7f1424071a80 2024-03-21T08:41:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f141401a070 con 0x7f1424071a80 2024-03-21T08:41:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f141aa00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 0x7f1400069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1414086260 con 0x7f1424071a80 2024-03-21T08:41:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f141be00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 0x7f1400069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.133+0000 7f141be00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 0x7f1400069fe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f140c0af000 tx=0x7f140c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.137+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1414056300 con 0x7f1424071a80 2024-03-21T08:41:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.305+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f142404f4f0 con 0x7f1424071a80 2024-03-21T08:41:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.305+0000 7f141aa00640 1 -- 172.21.15.28:0/4100289114 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f141405a3a0 con 0x7f1424071a80 2024-03-21T08:41:47.310 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 msgr2=0x7f1400069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 0x7f1400069fe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f140c0af000 tx=0x7f140c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 msgr2=0x7f142413f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f141400e930 tx=0x7f141400edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 shutdown_connections 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1400067b20 0x7f1400069fe0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f142415cbb0 0x7f14240b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14240cde40 0x7f1424157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 --2- 172.21.15.28:0/4100289114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1424071a80 0x7f142413f540 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 >> 172.21.15.28:0/4100289114 conn(0x7f14240674f0 msgr2=0x7f1424071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 shutdown_connections 2024-03-21T08:41:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.309+0000 7f1429a00640 1 -- 172.21.15.28:0/4100289114 wait complete. 2024-03-21T08:41:47.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 Processor -- start 2024-03-21T08:41:47.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 -- start start 2024-03-21T08:41:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 0x7fc5dc06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc5dc15cbb0 0x7fc5dc0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc0ce5b0 con 0x7fc5dc0cde40 2024-03-21T08:41:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc15c720 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.469+0000 7fc5e4800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc0ce890 con 0x7fc5dc071a80 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc5dc0cde40 0x7fc5dc071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56936/0 (socket says 172.21.15.28:56936) 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1400640 1 -- 172.21.15.28:0/396199040 learned_addr learned my addr 172.21.15.28:0/396199040 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/396199040 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 0x7fc5dc06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e2800640 1 --2- 172.21.15.28:0/396199040 >> v2:172.21.15.151:3300/0 conn(0x7fc5dc15cbb0 0x7fc5dc0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e0a00640 1 -- 172.21.15.28:0/396199040 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613682586 0 0) 0x7fc5dc0ce5b0 con 0x7fc5dc0cde40 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 -- 172.21.15.28:0/396199040 >> v2:172.21.15.151:3300/0 conn(0x7fc5dc15cbb0 msgr2=0x7fc5dc0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/396199040 >> v2:172.21.15.151:3300/0 conn(0x7fc5dc15cbb0 0x7fc5dc0cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 -- 172.21.15.28:0/396199040 >> v1:172.21.15.28:6789/0 conn(0x7fc5dc0cde40 legacy=0x7fc5dc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 -- 172.21.15.28:0/396199040 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc15c3b0 con 0x7fc5dc071a80 2024-03-21T08:41:47.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/396199040 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 0x7fc5dc06f0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00097c0 tx=0x7fc5d001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5456ba4ec65559 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e0a00640 1 -- 172.21.15.28:0/396199040 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00ab5f0 con 0x7fc5dc071a80 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e0a00640 1 -- 172.21.15.28:0/396199040 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d00b0070 con 0x7fc5dc071a80 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.473+0000 7fc5e0a00640 1 -- 172.21.15.28:0/396199040 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00ab9b0 con 0x7fc5dc071a80 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- 172.21.15.28:0/396199040 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 msgr2=0x7fc5dc06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- 172.21.15.28:0/396199040 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 0x7fc5dc06f0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00097c0 tx=0x7fc5d001c410 comp rx=0 tx=0).stop 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- 172.21.15.28:0/396199040 shutdown_connections 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- 172.21.15.28:0/396199040 >> v2:172.21.15.151:3300/0 conn(0x7fc5dc15cbb0 0x7fc5dc0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- 172.21.15.28:0/396199040 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc071a80 0x7fc5dc06f0a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- 172.21.15.28:0/396199040 >> 172.21.15.28:0/396199040 conn(0x7fc5dc0674f0 msgr2=0x7fc5dc06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- 172.21.15.28:0/396199040 shutdown_connections 2024-03-21T08:41:47.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- 172.21.15.28:0/396199040 wait complete. 2024-03-21T08:41:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 Processor -- start 2024-03-21T08:41:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- start start 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5dc071a80 0x7fc5dc13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc0cde40 0x7fc5dc0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc168f20 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc144b20 con 0x7fc5dc071a80 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e4800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc1690a0 con 0x7fc5dc0cde40 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35248/0 (socket says 172.21.15.28:35248) 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 -- 172.21.15.28:0/1738495626 learned_addr learned my addr 172.21.15.28:0/1738495626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e1400640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc0cde40 0x7fc5dc0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5dc071a80 0x7fc5dc13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 -- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc0cde40 msgr2=0x7fc5dc0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc0cde40 0x7fc5dc0ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 -- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5dc071a80 msgr2=0x7fc5dc13a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5dc071a80 0x7fc5dc13a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 -- 172.21.15.28:0/1738495626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc0ad790 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.477+0000 7fc5e2800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ec10 tx=0x7fc5d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8018020 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d800cd20 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80146c0 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc0ada70 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc0ae030 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc073430 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc5d801b020 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 0x7fc5c0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc5d80870c0 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.481+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 0x7fc5c0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.485+0000 7fc5e1e00640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 0x7fc5c0069fe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00af000 tx=0x7fc5d00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.485+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc5d8022030 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5dc06b570 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5d6000640 1 -- 172.21.15.28:0/1738495626 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5d8010040 con 0x7fc5dc15cbb0 2024-03-21T08:41:47.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 msgr2=0x7fc5c0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 0x7fc5c0069fe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00af000 tx=0x7fc5d00a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 msgr2=0x7fc5dc0af280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ec10 tx=0x7fc5d800c490 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 shutdown_connections 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5dc15cbb0 0x7fc5dc0af280 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc5c0067b20 0x7fc5c0069fe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc5dc0cde40 0x7fc5dc0ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 --2- 172.21.15.28:0/1738495626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc5dc071a80 0x7fc5dc13a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 >> 172.21.15.28:0/1738495626 conn(0x7fc5dc0674f0 msgr2=0x7fc5dc072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 shutdown_connections 2024-03-21T08:41:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.649+0000 7fc5e4800640 1 -- 172.21.15.28:0/1738495626 wait complete. 2024-03-21T08:41:47.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 Processor -- start 2024-03-21T08:41:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 -- start start 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f40cc071a80 0x7f40cc06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc0cde40 0x7f40cc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc15c720 con 0x7f40cc15cbb0 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40cc0723a0 con 0x7f40cc071a80 2024-03-21T08:41:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40d1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0ce5b0 con 0x7f40cc0cde40 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35256/0 (socket says 172.21.15.28:35256) 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 -- 172.21.15.28:0/1450186089 learned_addr learned my addr 172.21.15.28:0/1450186089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40c2000640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc0cde40 0x7f40cc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40caa00640 1 --2- 172.21.15.28:0/1450186089 >> v2:172.21.15.151:3300/0 conn(0x7f40cc071a80 0x7f40cc06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 -- 172.21.15.28:0/1450186089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc0cde40 msgr2=0x7f40cc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc0cde40 0x7f40cc071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 -- 172.21.15.28:0/1450186089 >> v2:172.21.15.151:3300/0 conn(0x7f40cc071a80 msgr2=0x7f40cc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 --2- 172.21.15.28:0/1450186089 >> v2:172.21.15.151:3300/0 conn(0x7f40cc071a80 0x7f40cc06f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.817+0000 7f40cb400640 1 -- 172.21.15.28:0/1450186089 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0ce890 con 0x7f40cc15cbb0 2024-03-21T08:41:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40cb400640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0b4f30 tx=0x7f40c401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f7c73d829678ba server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40ca000640 1 -- 172.21.15.28:0/1450186089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c40a57d0 con 0x7f40cc15cbb0 2024-03-21T08:41:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40ca000640 1 -- 172.21.15.28:0/1450186089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c40a5970 con 0x7f40cc15cbb0 2024-03-21T08:41:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40ca000640 1 -- 172.21.15.28:0/1450186089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c40a5d40 con 0x7f40cc15cbb0 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 -- 172.21.15.28:0/1450186089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc15cbb0 msgr2=0x7f40cc0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0b4f30 tx=0x7f40c401c930 comp rx=0 tx=0).stop 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 -- 172.21.15.28:0/1450186089 shutdown_connections 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc15cbb0 0x7f40cc0cc4b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1450186089 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc0cde40 0x7f40cc071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1450186089 >> v2:172.21.15.151:3300/0 conn(0x7f40cc071a80 0x7f40cc06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 -- 172.21.15.28:0/1450186089 >> 172.21.15.28:0/1450186089 conn(0x7f40cc0674f0 msgr2=0x7f40cc06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 -- 172.21.15.28:0/1450186089 shutdown_connections 2024-03-21T08:41:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.821+0000 7f40d1a00640 1 -- 172.21.15.28:0/1450186089 wait complete. 2024-03-21T08:41:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 Processor -- start 2024-03-21T08:41:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- start start 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 0x7f40cc148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc15cbb0 0x7f40cc0cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc0ce220 con 0x7f40cc071a80 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc144b20 con 0x7f40cc0cde40 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc06e2b0 con 0x7f40cc15cbb0 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35272/0 (socket says 172.21.15.28:35272) 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 -- 172.21.15.28:0/1096053086 learned_addr learned my addr 172.21.15.28:0/1096053086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40c2000640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 0x7f40cc148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40cb400640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc15cbb0 0x7f40cc0cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 -- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc15cbb0 msgr2=0x7f40cc0cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc15cbb0 0x7f40cc0cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 -- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 msgr2=0x7f40cc148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 0x7f40cc148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 -- 172.21.15.28:0/1096053086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc149560 con 0x7f40cc071a80 2024-03-21T08:41:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40c2000640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 0x7f40cc148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40caa00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e930 tx=0x7f40bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc017070 con 0x7f40cc071a80 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc00ca00 con 0x7f40cc071a80 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc00cdb0 con 0x7f40cc071a80 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40cc149840 con 0x7f40cc071a80 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.825+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40cc149e00 con 0x7f40cc071a80 2024-03-21T08:41:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40cc073430 con 0x7f40cc071a80 2024-03-21T08:41:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f40bc01a070 con 0x7f40cc071a80 2024-03-21T08:41:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40c3400640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 0x7f40a8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f40bc086270 con 0x7f40cc071a80 2024-03-21T08:41:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40c2000640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 0x7f40a8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.829+0000 7f40c2000640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 0x7f40a8069fe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40cc1fa6a0 tx=0x7f40b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.833+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f40bc056310 con 0x7f40cc071a80 2024-03-21T08:41:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.993+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f40cc04f4f0 con 0x7f40cc071a80 2024-03-21T08:41:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.993+0000 7f40c3400640 1 -- 172.21.15.28:0/1096053086 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f40bc05a3b0 con 0x7f40cc071a80 2024-03-21T08:41:47.999 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:48.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 msgr2=0x7f40a8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 0x7f40a8069fe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f40cc1fa6a0 tx=0x7f40b4009190 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 msgr2=0x7f40cc13b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e930 tx=0x7f40bc00edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 shutdown_connections 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f40a8067b20 0x7f40a8069fe0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f40cc15cbb0 0x7f40cc0cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f40cc0cde40 0x7f40cc148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 --2- 172.21.15.28:0/1096053086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40cc071a80 0x7f40cc13b2b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 >> 172.21.15.28:0/1096053086 conn(0x7f40cc0674f0 msgr2=0x7f40cc0cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 shutdown_connections 2024-03-21T08:41:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:47.997+0000 7f40d1a00640 1 -- 172.21.15.28:0/1096053086 wait complete. 2024-03-21T08:41:48.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.165+0000 7fcd2be00640 1 Processor -- start 2024-03-21T08:41:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.165+0000 7fcd2be00640 1 -- start start 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd2c06c490 0x7fcd2c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0d0380 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2be00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0d0660 con 0x7fcd2c09a140 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0d0150 con 0x7fcd2c06c490 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd2c0b0ae0 0x7fcd2c099a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56940/0 (socket says 172.21.15.28:56940) 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2aa00640 1 -- 172.21.15.28:0/1317913691 learned_addr learned my addr 172.21.15.28:0/1317913691 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2b400640 1 --2- 172.21.15.28:0/1317913691 >> v2:172.21.15.28:3300/0 conn(0x7fcd2c06c490 0x7fcd2c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833111086 0 0) 0x7fcd2c0d0380 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd10003530 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2b400640 1 -- 172.21.15.28:0/1317913691 >> v1:172.21.15.151:6789/0 conn(0x7fcd2c09a140 legacy=0x7fcd2c15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460657366 0 0) 0x7fcd2c0d0660 con 0x7fcd2c09a140 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2b400640 1 -- 172.21.15.28:0/1317913691 >> v1:172.21.15.28:6789/0 conn(0x7fcd2c0b0ae0 legacy=0x7fcd2c099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2b400640 1 -- 172.21.15.28:0/1317913691 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0d0940 con 0x7fcd2c06c490 2024-03-21T08:41:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2b400640 1 --2- 172.21.15.28:0/1317913691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c06c490 0x7fcd2c0cfc10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009d90 tx=0x7fcd2401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619bdd01f927534c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.169+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b2070 con 0x7fcd2c06c490 2024-03-21T08:41:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240a57a0 con 0x7fcd2c06c490 2024-03-21T08:41:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2a000640 1 -- 172.21.15.28:0/1317913691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a5b70 con 0x7fcd2c06c490 2024-03-21T08:41:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- 172.21.15.28:0/1317913691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c06c490 msgr2=0x7fcd2c0cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 --2- 172.21.15.28:0/1317913691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c06c490 0x7fcd2c0cfc10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009d90 tx=0x7fcd2401c8e0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- 172.21.15.28:0/1317913691 shutdown_connections 2024-03-21T08:41:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 --2- 172.21.15.28:0/1317913691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c06c490 0x7fcd2c0cfc10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- 172.21.15.28:0/1317913691 >> 172.21.15.28:0/1317913691 conn(0x7fcd2c0b3390 msgr2=0x7fcd2c0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- 172.21.15.28:0/1317913691 shutdown_connections 2024-03-21T08:41:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- 172.21.15.28:0/1317913691 wait complete. 2024-03-21T08:41:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 Processor -- start 2024-03-21T08:41:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- start start 2024-03-21T08:41:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd2c06c490 0x7fcd2c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd2c09a140 0x7fcd2c150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0cd080 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c0ccf00 con 0x7fcd2c09a140 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2be00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c15e030 con 0x7fcd2c06c490 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35290/0 (socket says 172.21.15.28:35290) 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2b400640 1 -- 172.21.15.28:0/671845767 learned_addr learned my addr 172.21.15.28:0/671845767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd2aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd2c06c490 0x7fcd2c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.173+0000 7fcd22000640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd2c09a140 0x7fcd2c150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 -- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd2c06c490 msgr2=0x7fcd2c154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd2c06c490 0x7fcd2c154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 -- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd2c09a140 msgr2=0x7fcd2c150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd2c09a140 0x7fcd2c150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 -- 172.21.15.28:0/671845767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c14f4b0 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2b400640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b0040 tx=0x7fcd24019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240bd020 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240b3040 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b2050 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c14f790 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c14fd50 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c075270 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcd240ad650 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 0x7fcd08069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fcd24120be0 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.177+0000 7fcd2aa00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 0x7fcd08069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.181+0000 7fcd2aa00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 0x7fcd08069fe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007fd0 tx=0x7fcd1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.181+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcd240c7020 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.349+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd2c06ff90 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.349+0000 7fcd23400640 1 -- 172.21.15.28:0/671845767 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18930 (secure 0 0 0) 0x7fcd24016560 con 0x7fcd2c0b0ae0 2024-03-21T08:41:48.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:48.354 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:48.134935+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 msgr2=0x7fcd08069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 0x7fcd08069fe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007fd0 tx=0x7fcd1c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 msgr2=0x7fcd2c147c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b0040 tx=0x7fcd24019680 comp rx=0 tx=0).stop 2024-03-21T08:41:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 shutdown_connections 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd2c0b0ae0 0x7fcd2c147c30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd2c09a140 0x7fcd2c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd08067b20 0x7fcd08069fe0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 --2- 172.21.15.28:0/671845767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd2c06c490 0x7fcd2c154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 >> 172.21.15.28:0/671845767 conn(0x7fcd2c0b3390 msgr2=0x7fcd2c070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 shutdown_connections 2024-03-21T08:41:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.353+0000 7fcd2be00640 1 -- 172.21.15.28:0/671845767 wait complete. 2024-03-21T08:41:48.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:48.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.513+0000 7fab65200640 1 Processor -- start 2024-03-21T08:41:48.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.513+0000 7fab65200640 1 -- start start 2024-03-21T08:41:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab65200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab600cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab65200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6006f500 con 0x7fab6015cbb0 2024-03-21T08:41:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab65200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600ce5b0 con 0x7fab60071a80 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab65200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600ce380 con 0x7fab6006f0a0 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fab6015cbb0 0x7fab600cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56948/0 (socket says 172.21.15.28:56948) 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 -- 172.21.15.28:0/1979869881 learned_addr learned my addr 172.21.15.28:0/1979869881 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab55400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab600cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743881857 0 0) 0x7fab6006f500 con 0x7fab6015cbb0 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab44003530 con 0x7fab6015cbb0 2024-03-21T08:41:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405069628 0 0) 0x7fab600ce5b0 con 0x7fab60071a80 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6006f500 con 0x7fab60071a80 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321187552 0 0) 0x7fab44003530 con 0x7fab6015cbb0 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 msgr2=0x7fab600cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab600cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 >> v1:172.21.15.151:6789/0 conn(0x7fab60071a80 legacy=0x7fab60071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600ce890 con 0x7fab6015cbb0 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fab58001680 con 0x7fab6015cbb0 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 >> v1:172.21.15.28:6789/0 conn(0x7fab6015cbb0 legacy=0x7fab600cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab44003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab44004130 0x7fab44004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab44004d50 0x7fab440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400bb60 con 0x7fab44004130 2024-03-21T08:41:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400bd10 con 0x7fab44004d50 2024-03-21T08:41:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4400bf10 con 0x7fab6006f0a0 2024-03-21T08:41:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab55400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab44004130 0x7fab44004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5de00640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab44003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab44004d50 0x7fab440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 -- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 msgr2=0x7fab44003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab6006f0a0 0x7fab44003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 -- 172.21.15.28:0/1979869881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab44004130 msgr2=0x7fab44004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab44004130 0x7fab44004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab5e800640 1 -- 172.21.15.28:0/1979869881 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab44003530 con 0x7fab44004d50 2024-03-21T08:41:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.517+0000 7fab55400640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab44004130 0x7fab44004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab5e800640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab44004d50 0x7fab440051d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab58007fd0 tx=0x7fab5801cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=471219b64e64e01b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58007bf0 con 0x7fab44004d50 2024-03-21T08:41:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580a9070 con 0x7fab44004d50 2024-03-21T08:41:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab5d400640 1 -- 172.21.15.28:0/1979869881 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58005d80 con 0x7fab44004d50 2024-03-21T08:41:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 -- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab44004d50 msgr2=0x7fab440051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 --2- 172.21.15.28:0/1979869881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab44004d50 0x7fab440051d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fab58007fd0 tx=0x7fab5801cb40 comp rx=0 tx=0).stop 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab5e800640 1 -- 172.21.15.28:0/1979869881 reap_dead start 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 -- 172.21.15.28:0/1979869881 shutdown_connections 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 -- 172.21.15.28:0/1979869881 >> 172.21.15.28:0/1979869881 conn(0x7fab600674f0 msgr2=0x7fab6006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 -- 172.21.15.28:0/1979869881 shutdown_connections 2024-03-21T08:41:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.521+0000 7fab65200640 1 -- 172.21.15.28:0/1979869881 wait complete. 2024-03-21T08:41:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 Processor -- start 2024-03-21T08:41:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- start start 2024-03-21T08:41:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab601f4fd0 0x7fab6013e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab601f5430 0x7fab6013ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600cdfc0 con 0x7fab60159a90 2024-03-21T08:41:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600cde40 con 0x7fab601f5430 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600ce140 con 0x7fab601f4fd0 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35306/0 (socket says 172.21.15.28:35306) 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 -- 172.21.15.28:0/103100357 learned_addr learned my addr 172.21.15.28:0/103100357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab55400640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab601f4fd0 0x7fab6013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5e800640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab601f5430 0x7fab6013ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 -- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab601f4fd0 msgr2=0x7fab6013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab601f4fd0 0x7fab6013e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 -- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab601f5430 msgr2=0x7fab6013ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab601f5430 0x7fab6013ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 -- 172.21.15.28:0/103100357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6015a750 con 0x7fab60159a90 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab5de00640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab5000fb50 tx=0x7fab5000ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50003b90 con 0x7fab60159a90 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab50018070 con 0x7fab60159a90 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5000d8b0 con 0x7fab60159a90 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab600b1130 con 0x7fab60159a90 2024-03-21T08:41:48.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab60141ac0 con 0x7fab60159a90 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.525+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60073430 con 0x7fab60159a90 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.529+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fab5000da50 con 0x7fab60159a90 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.529+0000 7fab56a00640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 0x7fab3c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.529+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fab50086660 con 0x7fab60159a90 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.529+0000 7fab55400640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 0x7fab3c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.529+0000 7fab55400640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 0x7fab3c069fe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab601409c0 tx=0x7fab48009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.533+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab50056b30 con 0x7fab60159a90 2024-03-21T08:41:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.697+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab600685a0 con 0x7fab60159a90 2024-03-21T08:41:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.697+0000 7fab56a00640 1 -- 172.21.15.28:0/103100357 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab50056430 con 0x7fab60159a90 2024-03-21T08:41:48.702 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 msgr2=0x7fab3c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 0x7fab3c069fe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab601409c0 tx=0x7fab48009190 comp rx=0 tx=0).stop 2024-03-21T08:41:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 msgr2=0x7fab600b0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab5000fb50 tx=0x7fab5000ff50 comp rx=0 tx=0).stop 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 shutdown_connections 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fab601f5430 0x7fab6013ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab3c067b20 0x7fab3c069fe0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fab601f4fd0 0x7fab6013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 --2- 172.21.15.28:0/103100357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab60159a90 0x7fab600b0bf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 >> 172.21.15.28:0/103100357 conn(0x7fab600674f0 msgr2=0x7fab6013c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 shutdown_connections 2024-03-21T08:41:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.701+0000 7fab65200640 1 -- 172.21.15.28:0/103100357 wait complete. 2024-03-21T08:41:48.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 Processor -- start 2024-03-21T08:41:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 -- start start 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a700b4420 0x7f5a70071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a700b4800 0x7f5a70071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7015bf90 con 0x7f5a700ccbf0 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7015b1d0 con 0x7f5a700b4420 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a76600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7015b4b0 con 0x7f5a700b4800 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6f400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5a700b4420 0x7f5a70071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a66a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a700b4800 0x7f5a70071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6fe00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f5a700ccbf0 0x7f5a700cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:34794/0 (socket says 172.21.15.28:34794) 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6fe00640 1 -- 172.21.15.28:0/1917188369 learned_addr learned my addr 172.21.15.28:0/1917188369 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6f400640 1 -- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a700b4800 msgr2=0x7f5a70071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6f400640 1 --2- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a700b4800 0x7f5a70071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6f400640 1 -- 172.21.15.28:0/1917188369 >> v1:172.21.15.151:6789/0 conn(0x7f5a700ccbf0 legacy=0x7f5a700cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.865+0000 7f5a6f400640 1 -- 172.21.15.28:0/1917188369 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7015b790 con 0x7f5a700b4420 2024-03-21T08:41:48.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a6f400640 1 --2- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a70071a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a600097c0 tx=0x7f5a6001e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b91e8c4c27205 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a6ea00640 1 -- 172.21.15.28:0/1917188369 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600ac5f0 con 0x7f5a700b4420 2024-03-21T08:41:48.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a6ea00640 1 -- 172.21.15.28:0/1917188369 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a600b1070 con 0x7f5a700b4420 2024-03-21T08:41:48.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a6ea00640 1 -- 172.21.15.28:0/1917188369 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a600ac9b0 con 0x7f5a700b4420 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 -- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 msgr2=0x7f5a70071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 --2- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a70071a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a600097c0 tx=0x7f5a6001e5f0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 -- 172.21.15.28:0/1917188369 shutdown_connections 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 --2- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a700b4800 0x7f5a70071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 --2- 172.21.15.28:0/1917188369 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a70071a80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 -- 172.21.15.28:0/1917188369 >> 172.21.15.28:0/1917188369 conn(0x7f5a700674f0 msgr2=0x7f5a7006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 -- 172.21.15.28:0/1917188369 shutdown_connections 2024-03-21T08:41:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 -- 172.21.15.28:0/1917188369 wait complete. 2024-03-21T08:41:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.869+0000 7f5a76600640 1 Processor -- start 2024-03-21T08:41:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- start start 2024-03-21T08:41:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a701f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a701465f0 0x7f5a70159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7006e640 con 0x7f5a700ccbf0 2024-03-21T08:41:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7006e4c0 con 0x7f5a700b4420 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7006e7c0 con 0x7f5a701465f0 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35310/0 (socket says 172.21.15.28:35310) 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 -- 172.21.15.28:0/3673833501 learned_addr learned my addr 172.21.15.28:0/3673833501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a6fe00640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a701465f0 0x7f5a70159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a6f400640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a701f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 -- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a701465f0 msgr2=0x7f5a70159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a701465f0 0x7f5a70159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 -- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 msgr2=0x7f5a701f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a701f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 -- 172.21.15.28:0/3673833501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a701f6e80 con 0x7f5a700ccbf0 2024-03-21T08:41:48.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a66a00640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e840 tx=0x7f5a5800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58017070 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5800ee90 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800cb60 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70155890 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70155e10 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.873+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70156530 con 0x7f5a700ccbf0 2024-03-21T08:41:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.877+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5a5801a070 con 0x7f5a700ccbf0 2024-03-21T08:41:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.877+0000 7f5a67e00640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 0x7f5a4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.877+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f5a58086180 con 0x7f5a700ccbf0 2024-03-21T08:41:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.877+0000 7f5a6f400640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 0x7f5a4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:48.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.877+0000 7f5a6f400640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 0x7f5a4c069fe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a600b0000 tx=0x7f5a600a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:48.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:48.881+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5a58056120 con 0x7f5a700ccbf0 2024-03-21T08:41:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.045+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5a70073430 con 0x7f5a700ccbf0 2024-03-21T08:41:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.045+0000 7f5a67e00640 1 -- 172.21.15.28:0/3673833501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5a5805a1c0 con 0x7f5a700ccbf0 2024-03-21T08:41:49.051 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:49.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 msgr2=0x7f5a4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 0x7f5a4c069fe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a600b0000 tx=0x7f5a600a9000 comp rx=0 tx=0).stop 2024-03-21T08:41:49.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 msgr2=0x7f5a7013a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e840 tx=0x7f5a5800ec40 comp rx=0 tx=0).stop 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 shutdown_connections 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a4c067b20 0x7f5a4c069fe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5a701465f0 0x7f5a70159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a700ccbf0 0x7f5a7013a170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 --2- 172.21.15.28:0/3673833501 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5a700b4420 0x7f5a701f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 >> 172.21.15.28:0/3673833501 conn(0x7f5a700674f0 msgr2=0x7f5a70152f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 shutdown_connections 2024-03-21T08:41:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.049+0000 7f5a76600640 1 -- 172.21.15.28:0/3673833501 wait complete. 2024-03-21T08:41:49.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.213+0000 7f065ea00640 1 Processor -- start 2024-03-21T08:41:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.213+0000 7f065ea00640 1 -- start start 2024-03-21T08:41:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 0x7f065806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06580ce5b0 con 0x7f065815cbb0 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ea00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06580ce890 con 0x7f06580cde40 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065815c720 con 0x7f0658071a80 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f065815cbb0 0x7f06580cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56956/0 (socket says 172.21.15.28:56956) 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f065ca00640 1 -- 172.21.15.28:0/2988252029 learned_addr learned my addr 172.21.15.28:0/2988252029 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657e00640 1 --2- 172.21.15.28:0/2988252029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 0x7f065806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417976403 0 0) 0x7f06580ce5b0 con 0x7f065815cbb0 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c003530 con 0x7f065815cbb0 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083022546 0 0) 0x7f06580ce890 con 0x7f06580cde40 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06580ce5b0 con 0x7f06580cde40 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657e00640 1 -- 172.21.15.28:0/2988252029 >> v1:172.21.15.151:6789/0 conn(0x7f06580cde40 legacy=0x7f0658071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657e00640 1 -- 172.21.15.28:0/2988252029 >> v1:172.21.15.28:6789/0 conn(0x7f065815cbb0 legacy=0x7f06580cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.217+0000 7f0657e00640 1 -- 172.21.15.28:0/2988252029 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065815c3b0 con 0x7f0658071a80 2024-03-21T08:41:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f0657e00640 1 --2- 172.21.15.28:0/2988252029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 0x7f065806f0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0648009dc0 tx=0x7f064801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6622559317f04c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480b2070 con 0x7f0658071a80 2024-03-21T08:41:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06480a59e0 con 0x7f0658071a80 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f0657400640 1 -- 172.21.15.28:0/2988252029 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480a5db0 con 0x7f0658071a80 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- 172.21.15.28:0/2988252029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 msgr2=0x7f065806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 --2- 172.21.15.28:0/2988252029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 0x7f065806f0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0648009dc0 tx=0x7f064801c900 comp rx=0 tx=0).stop 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- 172.21.15.28:0/2988252029 shutdown_connections 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 --2- 172.21.15.28:0/2988252029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0658071a80 0x7f065806f0a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- 172.21.15.28:0/2988252029 >> 172.21.15.28:0/2988252029 conn(0x7f06580674f0 msgr2=0x7f065806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- 172.21.15.28:0/2988252029 shutdown_connections 2024-03-21T08:41:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- 172.21.15.28:0/2988252029 wait complete. 2024-03-21T08:41:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 Processor -- start 2024-03-21T08:41:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.221+0000 7f065ea00640 1 -- start start 2024-03-21T08:41:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0658071a80 0x7f0658159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f065815cbb0 0x7f06580aaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065816a0b0 con 0x7f06580cde40 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0658169f30 con 0x7f0658071a80 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065816a230 con 0x7f065815cbb0 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f0657e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0658071a80 0x7f0658159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35316/0 (socket says 172.21.15.28:35316) 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 -- 172.21.15.28:0/1371973277 learned_addr learned my addr 172.21.15.28:0/1371973277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ca00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f065815cbb0 0x7f06580aaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 -- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f065815cbb0 msgr2=0x7f06580aaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f065815cbb0 0x7f06580aaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 -- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0658071a80 msgr2=0x7f0658159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0658071a80 0x7f0658159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 -- 172.21.15.28:0/1371973277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065813ef70 con 0x7f06580cde40 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f064f400640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f064000c500 tx=0x7f064000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400052d0 con 0x7f06580cde40 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0640005490 con 0x7f06580cde40 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065813f250 con 0x7f06580cde40 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640012850 con 0x7f06580cde40 2024-03-21T08:41:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065813f810 con 0x7f06580cde40 2024-03-21T08:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0658073430 con 0x7f06580cde40 2024-03-21T08:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.225+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f06400129f0 con 0x7f06580cde40 2024-03-21T08:41:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.229+0000 7f0654c00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 0x7f0634069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.229+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f0640085f00 con 0x7f06580cde40 2024-03-21T08:41:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.229+0000 7f0657e00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 0x7f0634069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.229+0000 7f0657e00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 0x7f0634069fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f06480af000 tx=0x7f06480a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.233+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0640055e20 con 0x7f06580cde40 2024-03-21T08:41:49.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.393+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f065804f4f0 con 0x7f06580cde40 2024-03-21T08:41:49.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.393+0000 7f0654c00640 1 -- 172.21.15.28:0/1371973277 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0640059ec0 con 0x7f06580cde40 2024-03-21T08:41:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 msgr2=0x7f0634069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 0x7f0634069fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f06480af000 tx=0x7f06480a7070 comp rx=0 tx=0).stop 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 msgr2=0x7f0658140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f064000c500 tx=0x7f064000c9c0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 shutdown_connections 2024-03-21T08:41:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0634067b20 0x7f0634069fe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f065815cbb0 0x7f06580aaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06580cde40 0x7f0658140da0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 --2- 172.21.15.28:0/1371973277 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0658071a80 0x7f0658159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 >> 172.21.15.28:0/1371973277 conn(0x7f06580674f0 msgr2=0x7f06580afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 shutdown_connections 2024-03-21T08:41:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.397+0000 7f065ea00640 1 -- 172.21.15.28:0/1371973277 wait complete. 2024-03-21T08:41:49.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.561+0000 7fa263e00640 1 Processor -- start 2024-03-21T08:41:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 -- start start 2024-03-21T08:41:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 0x7fa2640b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa2640cc170 0x7fa264071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26415be30 con 0x7fa2640ce320 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2640ccea0 con 0x7fa2640cc170 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640cc8e0 con 0x7fa26415ba50 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa2640ce320 0x7fa2640b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56968/0 (socket says 172.21.15.28:56968) 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa263400640 1 -- 172.21.15.28:0/2378420681 learned_addr learned my addr 172.21.15.28:0/2378420681 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262a00640 1 --2- 172.21.15.28:0/2378420681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 0x7fa2640b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa25be00640 1 --2- 172.21.15.28:0/2378420681 >> v2:172.21.15.151:3300/0 conn(0x7fa2640cc170 0x7fa264071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262000640 1 -- 172.21.15.28:0/2378420681 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515969421 0 0) 0x7fa26415be30 con 0x7fa2640ce320 2024-03-21T08:41:49.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262000640 1 -- 172.21.15.28:0/2378420681 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa248003530 con 0x7fa2640ce320 2024-03-21T08:41:49.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262a00640 1 -- 172.21.15.28:0/2378420681 >> v2:172.21.15.151:3300/0 conn(0x7fa2640cc170 msgr2=0x7fa264071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:49.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262a00640 1 --2- 172.21.15.28:0/2378420681 >> v2:172.21.15.151:3300/0 conn(0x7fa2640cc170 0x7fa264071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262a00640 1 -- 172.21.15.28:0/2378420681 >> v1:172.21.15.28:6789/0 conn(0x7fa2640ce320 legacy=0x7fa2640b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.565+0000 7fa262a00640 1 -- 172.21.15.28:0/2378420681 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2640ccbc0 con 0x7fa26415ba50 2024-03-21T08:41:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa262a00640 1 --2- 172.21.15.28:0/2378420681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 0x7fa2640b0da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=459945490cc3b017 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa262000640 1 -- 172.21.15.28:0/2378420681 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540a7070 con 0x7fa26415ba50 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa262000640 1 -- 172.21.15.28:0/2378420681 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2540b0070 con 0x7fa26415ba50 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa262000640 1 -- 172.21.15.28:0/2378420681 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540ab890 con 0x7fa26415ba50 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- 172.21.15.28:0/2378420681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 msgr2=0x7fa2640b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 --2- 172.21.15.28:0/2378420681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 0x7fa2640b0da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097f0 tx=0x7fa25401c470 comp rx=0 tx=0).stop 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- 172.21.15.28:0/2378420681 shutdown_connections 2024-03-21T08:41:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 --2- 172.21.15.28:0/2378420681 >> v2:172.21.15.151:3300/0 conn(0x7fa2640cc170 0x7fa264071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 --2- 172.21.15.28:0/2378420681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa26415ba50 0x7fa2640b0da0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- 172.21.15.28:0/2378420681 >> 172.21.15.28:0/2378420681 conn(0x7fa26406fa90 msgr2=0x7fa26406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- 172.21.15.28:0/2378420681 shutdown_connections 2024-03-21T08:41:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- 172.21.15.28:0/2378420681 wait complete. 2024-03-21T08:41:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 Processor -- start 2024-03-21T08:41:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.569+0000 7fa263e00640 1 -- start start 2024-03-21T08:41:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2640cc170 0x7fa2640b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa26415ba50 0x7fa2640cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640cc550 con 0x7fa2640ce320 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640720f0 con 0x7fa26415ba50 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640ceb40 con 0x7fa2640cc170 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35324/0 (socket says 172.21.15.28:35324) 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 -- 172.21.15.28:0/3014768191 learned_addr learned my addr 172.21.15.28:0/3014768191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa262a00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2640cc170 0x7fa2640b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263400640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa26415ba50 0x7fa2640cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 -- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2640cc170 msgr2=0x7fa2640b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2640cc170 0x7fa2640b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 -- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa26415ba50 msgr2=0x7fa2640cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa26415ba50 0x7fa2640cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 -- 172.21.15.28:0/3014768191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2640cb5d0 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25be00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00e930 tx=0x7fa24c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c017070 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c00ca00 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa264155d90 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00cdb0 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa264156350 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2641553b0 con 0x7fa2640ce320 2024-03-21T08:41:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.573+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa24c01a070 con 0x7fa2640ce320 2024-03-21T08:41:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.577+0000 7fa25aa00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 0x7fa240069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.577+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fa24c086680 con 0x7fa2640ce320 2024-03-21T08:41:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.577+0000 7fa262a00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 0x7fa240069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.577+0000 7fa262a00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 0x7fa240069fe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2540af000 tx=0x7fa2540b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.581+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa24c0566f0 con 0x7fa2640ce320 2024-03-21T08:41:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.745+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa26404f410 con 0x7fa2640ce320 2024-03-21T08:41:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.745+0000 7fa25aa00640 1 -- 172.21.15.28:0/3014768191 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa24c05a790 con 0x7fa2640ce320 2024-03-21T08:41:49.752 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 msgr2=0x7fa240069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 0x7fa240069fe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2540af000 tx=0x7fa2540b6040 comp rx=0 tx=0).stop 2024-03-21T08:41:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 msgr2=0x7fa2640caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00e930 tx=0x7fa24c00edf0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 shutdown_connections 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa26415ba50 0x7fa2640cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2640ce320 0x7fa2640caa60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa240067b20 0x7fa240069fe0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 --2- 172.21.15.28:0/3014768191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2640cc170 0x7fa2640b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 >> 172.21.15.28:0/3014768191 conn(0x7fa26406fa90 msgr2=0x7fa2640c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 shutdown_connections 2024-03-21T08:41:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.749+0000 7fa263e00640 1 -- 172.21.15.28:0/3014768191 wait complete. 2024-03-21T08:41:49.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 Processor -- start 2024-03-21T08:41:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 -- start start 2024-03-21T08:41:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d015dd00 0x7f65d00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 0x7f65d006b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f65d006eb30 0x7f65d006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00d0540 con 0x7f65d006eb30 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d00cd6e0 con 0x7f65d00aff70 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65d4e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00cd9c0 con 0x7f65d015dd00 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f65d006eb30 0x7f65d006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65cde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d015dd00 0x7f65d00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65c5400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 0x7f65d006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f65d006eb30 0x7f65d006b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35330/0 (socket says 172.21.15.28:35330) 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 -- 172.21.15.28:0/3951015463 learned_addr learned my addr 172.21.15.28:0/3951015463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 -- 172.21.15.28:0/3951015463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d015dd00 msgr2=0x7f65d00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 --2- 172.21.15.28:0/3951015463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d015dd00 0x7f65d00cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 -- 172.21.15.28:0/3951015463 >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 msgr2=0x7f65d006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 --2- 172.21.15.28:0/3951015463 >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 0x7f65d006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65ce800640 1 -- 172.21.15.28:0/3951015463 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00cdca0 con 0x7f65d006eb30 2024-03-21T08:41:49.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.909+0000 7f65c5400640 1 --2- 172.21.15.28:0/3951015463 >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 0x7f65d006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65ce800640 1 --2- 172.21.15.28:0/3951015463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d006eb30 0x7f65d006b6b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65d00d1950 tx=0x7f65c801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76215c2b37fdcaa server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65cd400640 1 -- 172.21.15.28:0/3951015463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80aa430 con 0x7f65d006eb30 2024-03-21T08:41:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65cd400640 1 -- 172.21.15.28:0/3951015463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c80af070 con 0x7f65d006eb30 2024-03-21T08:41:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65cd400640 1 -- 172.21.15.28:0/3951015463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80aa7f0 con 0x7f65d006eb30 2024-03-21T08:41:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 -- 172.21.15.28:0/3951015463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d006eb30 msgr2=0x7f65d006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3951015463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d006eb30 0x7f65d006b6b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65d00d1950 tx=0x7f65c801c4f0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 -- 172.21.15.28:0/3951015463 shutdown_connections 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3951015463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d006eb30 0x7f65d006b6b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3951015463 >> v2:172.21.15.151:3300/0 conn(0x7f65d00aff70 0x7f65d006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3951015463 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d015dd00 0x7f65d00cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 -- 172.21.15.28:0/3951015463 >> 172.21.15.28:0/3951015463 conn(0x7f65d0099a30 msgr2=0x7f65d0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 -- 172.21.15.28:0/3951015463 shutdown_connections 2024-03-21T08:41:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.913+0000 7f65d4e00640 1 -- 172.21.15.28:0/3951015463 wait complete. 2024-03-21T08:41:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 Processor -- start 2024-03-21T08:41:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- start start 2024-03-21T08:41:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d006eb30 0x7f65d0153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 0x7f65d00c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00d1400 con 0x7f65d015dd00 2024-03-21T08:41:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00d1280 con 0x7f65d00aff70 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00d1580 con 0x7f65d006eb30 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35338/0 (socket says 172.21.15.28:35338) 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 -- 172.21.15.28:0/3448890605 learned_addr learned my addr 172.21.15.28:0/3448890605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65c5400640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 0x7f65d00c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65cde00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d006eb30 0x7f65d0153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 -- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d006eb30 msgr2=0x7f65d0153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d006eb30 0x7f65d0153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 -- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 msgr2=0x7f65d00c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 0x7f65d00c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 -- 172.21.15.28:0/3448890605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0155f90 con 0x7f65d015dd00 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65c5400640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 0x7f65d00c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65ce800640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f65c801cd90 tx=0x7f65c801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80bd3f0 con 0x7f65d015dd00 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d015e2f0 con 0x7f65d015dd00 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80bd7d0 con 0x7f65d015dd00 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d015b5a0 con 0x7f65d015dd00 2024-03-21T08:41:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.917+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d015bb60 con 0x7f65d015dd00 2024-03-21T08:41:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d004f410 con 0x7f65d015dd00 2024-03-21T08:41:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f65c80b33f0 con 0x7f65d015dd00 2024-03-21T08:41:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65c6a00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 0x7f65ac06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f65c8120100 con 0x7f65d015dd00 2024-03-21T08:41:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65cde00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 0x7f65ac06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.921+0000 7f65cde00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 0x7f65ac06a010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65c00113e0 tx=0x7f65c000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:49.925+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f65c8016110 con 0x7f65d015dd00 2024-03-21T08:41:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.093+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65d00cd4c0 con 0x7f65d015dd00 2024-03-21T08:41:50.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.093+0000 7f65c6a00640 1 -- 172.21.15.28:0/3448890605 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f65c80f0010 con 0x7f65d015dd00 2024-03-21T08:41:50.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:50.100 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 msgr2=0x7f65ac06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 0x7f65ac06a010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65c00113e0 tx=0x7f65c000b040 comp rx=0 tx=0).stop 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 msgr2=0x7f65d00c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f65c801cd90 tx=0x7f65c801ad30 comp rx=0 tx=0).stop 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 shutdown_connections 2024-03-21T08:41:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d015dd00 0x7f65d00c8540 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f65d00aff70 0x7f65d00c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65ac067b50 0x7f65ac06a010 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 --2- 172.21.15.28:0/3448890605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f65d006eb30 0x7f65d0153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 >> 172.21.15.28:0/3448890605 conn(0x7f65d0099a30 msgr2=0x7f65d00c4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 shutdown_connections 2024-03-21T08:41:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.097+0000 7f65d4e00640 1 -- 172.21.15.28:0/3448890605 wait complete. 2024-03-21T08:41:50.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:50.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 Processor -- start 2024-03-21T08:41:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 -- start start 2024-03-21T08:41:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb2c00cc640 0x7fb2c00cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00ccbd0 con 0x7fb2c015bdc0 2024-03-21T08:41:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00cceb0 con 0x7fb2c0099c90 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c7800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c00cd7e0 con 0x7fb2c00cc640 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2c015bdc0 0x7fb2c00d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56980/0 (socket says 172.21.15.28:56980) 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 -- 172.21.15.28:0/1569186557 learned_addr learned my addr 172.21.15.28:0/1569186557 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/1569186557 >> v2:172.21.15.151:3300/0 conn(0x7fb2c00cc640 0x7fb2c00cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876814028 0 0) 0x7fb2c00ccbd0 con 0x7fb2c015bdc0 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8003530 con 0x7fb2c015bdc0 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956786572 0 0) 0x7fb2c00cceb0 con 0x7fb2c0099c90 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c00ccbd0 con 0x7fb2c0099c90 2024-03-21T08:41:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550261056 0 0) 0x7fb2a8003530 con 0x7fb2c015bdc0 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 >> v2:172.21.15.151:3300/0 conn(0x7fb2c00cc640 msgr2=0x7fb2c00cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 --2- 172.21.15.28:0/1569186557 >> v2:172.21.15.151:3300/0 conn(0x7fb2c00cc640 0x7fb2c00cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 >> v1:172.21.15.151:6789/0 conn(0x7fb2c0099c90 legacy=0x7fb2c00b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00cd190 con 0x7fb2c015bdc0 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fb2b4001680 con 0x7fb2c015bdc0 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 >> v1:172.21.15.28:6789/0 conn(0x7fb2c015bdc0 legacy=0x7fb2c00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 0x7fb2a8003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8004810 0x7fb2a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8005220 0x7fb2a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00cd7e0 con 0x7fb2a8003ba0 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a800c150 con 0x7fb2a8004810 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a800c350 con 0x7fb2a8005220 2024-03-21T08:41:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 0x7fb2a8003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8004810 0x7fb2a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c5800640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8005220 0x7fb2a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 -- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8005220 msgr2=0x7fb2a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8005220 0x7fb2a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 -- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8004810 msgr2=0x7fb2a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8004810 0x7fb2a8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c4e00640 1 -- 172.21.15.28:0/1569186557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8003530 con 0x7fb2a8003ba0 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2a8004810 0x7fb2a8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.261+0000 7fb2c5800640 1 -- 172.21.15.28:0/1569186557 reap_dead start 2024-03-21T08:41:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 0x7fb2a8003fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400c0e0 tx=0x7fb2b401cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a75d59752721b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4007c50 con 0x7fb2a8003ba0 2024-03-21T08:41:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b40a9070 con 0x7fb2a8003ba0 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2bf400640 1 -- 172.21.15.28:0/1569186557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4005da0 con 0x7fb2a8003ba0 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 -- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 msgr2=0x7fb2a8003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 0x7fb2a8003fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400c0e0 tx=0x7fb2b401cb30 comp rx=0 tx=0).stop 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 -- 172.21.15.28:0/1569186557 shutdown_connections 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 --2- 172.21.15.28:0/1569186557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a8003ba0 0x7fb2a8003fa0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 -- 172.21.15.28:0/1569186557 >> 172.21.15.28:0/1569186557 conn(0x7fb2c00b3620 msgr2=0x7fb2c00b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 -- 172.21.15.28:0/1569186557 shutdown_connections 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.265+0000 7fb2c7800640 1 -- 172.21.15.28:0/1569186557 wait complete. 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 Processor -- start 2024-03-21T08:41:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- start start 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8003ba0 0x7fb2c00b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0151940 0x7fb2c01facb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c009a600 con 0x7fb2c00c8c00 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c009a480 con 0x7fb2c0151940 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00b1330 con 0x7fb2a8003ba0 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35360/0 (socket says 172.21.15.28:35360) 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 -- 172.21.15.28:0/971667514 learned_addr learned my addr 172.21.15.28:0/971667514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8003ba0 0x7fb2c00b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c5800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0151940 0x7fb2c01facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 -- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8003ba0 msgr2=0x7fb2c00b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8003ba0 0x7fb2c00b6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 -- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0151940 msgr2=0x7fb2c01facb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0151940 0x7fb2c01facb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 -- 172.21.15.28:0/971667514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00b56b0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0118d0 tx=0x7fb2ac011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac004220 con 0x7fb2c00c8c00 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac0043c0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac00fbd0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c015d2e0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.269+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c015d8a0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c009faf0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb2ac003620 con 0x7fb2c00c8c00 2024-03-21T08:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2bcc00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 0x7fb2a0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fb2ac086a40 con 0x7fb2c00c8c00 2024-03-21T08:41:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 0x7fb2a0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.273+0000 7fb2c4e00640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 0x7fb2a0069fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40ab000 tx=0x7fb2b40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.277+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb2ac019030 con 0x7fb2c00c8c00 2024-03-21T08:41:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2c004f4f0 con 0x7fb2c00c8c00 2024-03-21T08:41:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2bcc00640 1 -- 172.21.15.28:0/971667514 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2ac009e20 con 0x7fb2c00c8c00 2024-03-21T08:41:50.445 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 msgr2=0x7fb2a0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 0x7fb2a0069fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40ab000 tx=0x7fb2b40a7070 comp rx=0 tx=0).stop 2024-03-21T08:41:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 msgr2=0x7fb2c01fa770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0118d0 tx=0x7fb2ac011d90 comp rx=0 tx=0).stop 2024-03-21T08:41:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 shutdown_connections 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb2c0151940 0x7fb2c01facb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c00c8c00 0x7fb2c01fa770 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb2a0067b20 0x7fb2a0069fe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 --2- 172.21.15.28:0/971667514 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb2a8003ba0 0x7fb2c00b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 >> 172.21.15.28:0/971667514 conn(0x7fb2c00b3620 msgr2=0x7fb2c0071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 shutdown_connections 2024-03-21T08:41:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.441+0000 7fb2c7800640 1 -- 172.21.15.28:0/971667514 wait complete. 2024-03-21T08:41:50.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 Processor -- start 2024-03-21T08:41:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 -- start start 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 0x7fdbe00d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbe015e190 0x7fdbe00b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 0x7fdbe00af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe00d0c40 con 0x7fdbe015e190 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe00b42a0 con 0x7fdbe00d0320 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbe7a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe006bd40 con 0x7fdbe00d0ee0 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbe015e190 0x7fdbe00b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbe015e190 0x7fdbe00b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36652/0 (socket says 172.21.15.28:36652) 2024-03-21T08:41:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.605+0000 7fdbd7e00640 1 -- 172.21.15.28:0/3037452783 learned_addr learned my addr 172.21.15.28:0/3037452783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe5000640 1 --2- 172.21.15.28:0/3037452783 >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 0x7fdbe00d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe5a00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 0x7fdbe00af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 -- 172.21.15.28:0/3037452783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 msgr2=0x7fdbe00af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 0x7fdbe00af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 -- 172.21.15.28:0/3037452783 >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 msgr2=0x7fdbe00d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3037452783 >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 0x7fdbe00d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 -- 172.21.15.28:0/3037452783 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe006c020 con 0x7fdbe015e190 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe5a00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 0x7fdbe00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe5000640 1 --2- 172.21.15.28:0/3037452783 >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 0x7fdbe00d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe015e190 0x7fdbe00b3d60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0073810 tx=0x7fdbc801c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76f42c67558b0d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbdfe00640 1 -- 172.21.15.28:0/3037452783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80b0070 con 0x7fdbe015e190 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbdfe00640 1 -- 172.21.15.28:0/3037452783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc80ab600 con 0x7fdbe015e190 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbdfe00640 1 -- 172.21.15.28:0/3037452783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80ab9d0 con 0x7fdbe015e190 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3037452783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe015e190 msgr2=0x7fdbe00b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe015e190 0x7fdbe00b3d60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0073810 tx=0x7fdbc801c1d0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.609+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3037452783 shutdown_connections 2024-03-21T08:41:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe00d0ee0 0x7fdbe00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3037452783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe015e190 0x7fdbe00b3d60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3037452783 >> v2:172.21.15.151:3300/0 conn(0x7fdbe00d0320 0x7fdbe00d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3037452783 >> 172.21.15.28:0/3037452783 conn(0x7fdbe0099a30 msgr2=0x7fdbe00b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3037452783 shutdown_connections 2024-03-21T08:41:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3037452783 wait complete. 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 Processor -- start 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- start start 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe00d0ee0 0x7fdbe014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 0x7fdbe014add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00ad640 con 0x7fdbe00d0320 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00ad4c0 con 0x7fdbe00d0ee0 2024-03-21T08:41:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00ad7c0 con 0x7fdbe015e190 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36660/0 (socket says 172.21.15.28:36660) 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 -- 172.21.15.28:0/3663763379 learned_addr learned my addr 172.21.15.28:0/3663763379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe00d0ee0 0x7fdbe014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 0x7fdbe014add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 -- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 msgr2=0x7fdbe014add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 0x7fdbe014add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 -- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe00d0ee0 msgr2=0x7fdbe014a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe00d0ee0 0x7fdbe014a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 -- 172.21.15.28:0/3663763379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe015fa20 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 0x7fdbe014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe5000640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000e810 tx=0x7fdbd000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd000f040 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0004030 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe014c1b0 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.613+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0004400 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe0159f10 con 0x7fdbe00d0320 2024-03-21T08:41:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe0159d00 con 0x7fdbe00d0320 2024-03-21T08:41:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fdbd000c6d0 con 0x7fdbe00d0320 2024-03-21T08:41:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbdd600640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 0x7fdbbc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fdbd0085f60 con 0x7fdbe00d0320 2024-03-21T08:41:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 0x7fdbbc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.617+0000 7fdbd7e00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 0x7fdbbc069fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80ad000 tx=0x7fdbc80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.621+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdbd0017020 con 0x7fdbe00d0320 2024-03-21T08:41:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.785+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdbe00b4080 con 0x7fdbe00d0320 2024-03-21T08:41:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.785+0000 7fdbdd600640 1 -- 172.21.15.28:0/3663763379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdbd0055e70 con 0x7fdbe00d0320 2024-03-21T08:41:50.792 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 msgr2=0x7fdbbc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 0x7fdbbc069fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbc80ad000 tx=0x7fdbc80a6000 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 msgr2=0x7fdbe006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000e810 tx=0x7fdbd000ecd0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 shutdown_connections 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdbbc067b20 0x7fdbbc069fe0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdbe015e190 0x7fdbe014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdbe00d0ee0 0x7fdbe014a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 --2- 172.21.15.28:0/3663763379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbe00d0320 0x7fdbe006ea70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 >> 172.21.15.28:0/3663763379 conn(0x7fdbe0099a30 msgr2=0x7fdbe00aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 shutdown_connections 2024-03-21T08:41:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.789+0000 7fdbe7a00640 1 -- 172.21.15.28:0/3663763379 wait complete. 2024-03-21T08:41:50.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:50.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 Processor -- start 2024-03-21T08:41:50.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 -- start start 2024-03-21T08:41:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f71ac0cde40 0x7f71ac071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0723a0 con 0x7f71ac15cbb0 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac15c720 con 0x7f71ac0cde40 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b4400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0ce5b0 con 0x7f71ac071a80 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:55308/0 (socket says 172.21.15.28:55308) 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b1a00640 1 -- 172.21.15.28:0/2136903733 learned_addr learned my addr 172.21.15.28:0/2136903733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b2400640 1 --1- 172.21.15.28:0/2136903733 >> v1:172.21.15.151:6789/0 conn(0x7f71ac15cbb0 0x7f71ac0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:58928/0 (socket says 172.21.15.28:58928) 2024-03-21T08:41:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.953+0000 7f71b1000640 1 --2- 172.21.15.28:0/2136903733 >> v2:172.21.15.151:3300/0 conn(0x7f71ac0cde40 0x7f71ac071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1000640 1 -- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 msgr2=0x7f71ac06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1000640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1000640 1 -- 172.21.15.28:0/2136903733 >> v1:172.21.15.151:6789/0 conn(0x7f71ac15cbb0 legacy=0x7f71ac0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1000640 1 -- 172.21.15.28:0/2136903733 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0ce890 con 0x7f71ac0cde40 2024-03-21T08:41:50.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71abe00640 1 -- 172.21.15.28:0/2136903733 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883555564 0 0) 0x7f71ac0723a0 con 0x7f71ac15cbb0 2024-03-21T08:41:50.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1a00640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b1000640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac0cde40 0x7f71ac071e60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f71980097c0 tx=0x7f719801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72688b5c918a5bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71abe00640 1 -- 172.21.15.28:0/2136903733 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980ab600 con 0x7f71ac0cde40 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71abe00640 1 -- 172.21.15.28:0/2136903733 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71980b0070 con 0x7f71ac0cde40 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71abe00640 1 -- 172.21.15.28:0/2136903733 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980ab9c0 con 0x7f71ac0cde40 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b4400640 1 -- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac0cde40 msgr2=0x7f71ac071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.957+0000 7f71b4400640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac0cde40 0x7f71ac071e60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f71980097c0 tx=0x7f719801c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- 172.21.15.28:0/2136903733 shutdown_connections 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac0cde40 0x7f71ac071e60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 --2- 172.21.15.28:0/2136903733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac071a80 0x7f71ac06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- 172.21.15.28:0/2136903733 >> 172.21.15.28:0/2136903733 conn(0x7f71ac0674f0 msgr2=0x7f71ac06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- 172.21.15.28:0/2136903733 shutdown_connections 2024-03-21T08:41:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- 172.21.15.28:0/2136903733 wait complete. 2024-03-21T08:41:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 Processor -- start 2024-03-21T08:41:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- start start 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac071a80 0x7f71ac13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac0cde40 0x7f71ac1422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac168f20 con 0x7f71ac15cbb0 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac144b20 con 0x7f71ac071a80 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b4400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac1690a0 con 0x7f71ac0cde40 2024-03-21T08:41:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b1a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac071a80 0x7f71ac13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b1000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac0cde40 0x7f71ac1422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36676/0 (socket says 172.21.15.28:36676) 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 -- 172.21.15.28:0/2495426713 learned_addr learned my addr 172.21.15.28:0/2495426713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 -- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac0cde40 msgr2=0x7f71ac1422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac0cde40 0x7f71ac1422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 -- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac071a80 msgr2=0x7f71ac13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac071a80 0x7f71ac13ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 -- 172.21.15.28:0/2495426713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0b1850 con 0x7f71ac15cbb0 2024-03-21T08:41:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71b2400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c780 tx=0x7f71a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.961+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4018070 con 0x7f71ac15cbb0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4004d70 con 0x7f71ac15cbb0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac0b1b30 con 0x7f71ac15cbb0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a400a880 con 0x7f71ac15cbb0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac0b20f0 con 0x7f71ac15cbb0 2024-03-21T08:41:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac073430 con 0x7f71ac15cbb0 2024-03-21T08:41:50.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f71a401b070 con 0x7f71ac15cbb0 2024-03-21T08:41:50.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71a9600640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 0x7f718c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:50.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f71a40858e0 con 0x7f71ac15cbb0 2024-03-21T08:41:50.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71b1a00640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 0x7f718c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:50.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.965+0000 7f71b1a00640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 0x7f718c069fe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71a0006cb0 tx=0x7f71a0009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:50.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:50.969+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f71a40557f0 con 0x7f71ac15cbb0 2024-03-21T08:41:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71ac04f4f0 con 0x7f71ac15cbb0 2024-03-21T08:41:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71a9600640 1 -- 172.21.15.28:0/2495426713 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71a4059890 con 0x7f71ac15cbb0 2024-03-21T08:41:51.137 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 msgr2=0x7f718c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 0x7f718c069fe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71a0006cb0 tx=0x7f71a0009240 comp rx=0 tx=0).stop 2024-03-21T08:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 msgr2=0x7f71ac15a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c780 tx=0x7f71a400cc40 comp rx=0 tx=0).stop 2024-03-21T08:41:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 shutdown_connections 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac15cbb0 0x7f71ac15a3a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f718c067b20 0x7f718c069fe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71ac0cde40 0x7f71ac1422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 --2- 172.21.15.28:0/2495426713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71ac071a80 0x7f71ac13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 >> 172.21.15.28:0/2495426713 conn(0x7f71ac0674f0 msgr2=0x7f71ac072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 shutdown_connections 2024-03-21T08:41:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.133+0000 7f71b4400640 1 -- 172.21.15.28:0/2495426713 wait complete. 2024-03-21T08:41:51.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.297+0000 7fc92dc00640 1 Processor -- start 2024-03-21T08:41:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.297+0000 7fc92dc00640 1 -- start start 2024-03-21T08:41:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 0x7fc92806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc92dc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc92815cbb0 0x7fc9280cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc92dc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280ce5b0 con 0x7fc9280cde40 2024-03-21T08:41:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc92dc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92815c720 con 0x7fc92815cbb0 2024-03-21T08:41:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc92dc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280ce890 con 0x7fc928071a80 2024-03-21T08:41:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 0x7fc92806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926000640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fc9280cde40 0x7fc928071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:58934/0 (socket says 172.21.15.28:58934) 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926000640 1 -- 172.21.15.28:0/1755504388 learned_addr learned my addr 172.21.15.28:0/1755504388 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc927400640 1 --2- 172.21.15.28:0/1755504388 >> v2:172.21.15.151:3300/0 conn(0x7fc92815cbb0 0x7fc9280cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926a00640 1 -- 172.21.15.28:0/1755504388 >> v2:172.21.15.151:3300/0 conn(0x7fc92815cbb0 msgr2=0x7fc9280cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc925600640 1 -- 172.21.15.28:0/1755504388 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835484101 0 0) 0x7fc9280ce5b0 con 0x7fc9280cde40 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926a00640 1 --2- 172.21.15.28:0/1755504388 >> v2:172.21.15.151:3300/0 conn(0x7fc92815cbb0 0x7fc9280cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926a00640 1 -- 172.21.15.28:0/1755504388 >> v1:172.21.15.151:6789/0 conn(0x7fc9280cde40 legacy=0x7fc928071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.301+0000 7fc926a00640 1 -- 172.21.15.28:0/1755504388 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92815c3b0 con 0x7fc928071a80 2024-03-21T08:41:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc926a00640 1 --2- 172.21.15.28:0/1755504388 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 0x7fc92806f0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9180097c0 tx=0x7fc91801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40efe35d86e358f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc925600640 1 -- 172.21.15.28:0/1755504388 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180ab5f0 con 0x7fc928071a80 2024-03-21T08:41:51.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc925600640 1 -- 172.21.15.28:0/1755504388 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9180b0070 con 0x7fc928071a80 2024-03-21T08:41:51.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc925600640 1 -- 172.21.15.28:0/1755504388 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180ab9b0 con 0x7fc928071a80 2024-03-21T08:41:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- 172.21.15.28:0/1755504388 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 msgr2=0x7fc92806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 --2- 172.21.15.28:0/1755504388 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 0x7fc92806f0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9180097c0 tx=0x7fc91801c410 comp rx=0 tx=0).stop 2024-03-21T08:41:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- 172.21.15.28:0/1755504388 shutdown_connections 2024-03-21T08:41:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 --2- 172.21.15.28:0/1755504388 >> v2:172.21.15.151:3300/0 conn(0x7fc92815cbb0 0x7fc9280cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 --2- 172.21.15.28:0/1755504388 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc928071a80 0x7fc92806f0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- 172.21.15.28:0/1755504388 >> 172.21.15.28:0/1755504388 conn(0x7fc9280674f0 msgr2=0x7fc92806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- 172.21.15.28:0/1755504388 shutdown_connections 2024-03-21T08:41:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- 172.21.15.28:0/1755504388 wait complete. 2024-03-21T08:41:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 Processor -- start 2024-03-21T08:41:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.305+0000 7fc92dc00640 1 -- start start 2024-03-21T08:41:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280cde40 0x7fc9280ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc92815cbb0 0x7fc92815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928168f20 con 0x7fc928071a80 2024-03-21T08:41:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928144b20 con 0x7fc9280cde40 2024-03-21T08:41:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9281690a0 con 0x7fc92815cbb0 2024-03-21T08:41:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36690/0 (socket says 172.21.15.28:36690) 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 -- 172.21.15.28:0/2573381148 learned_addr learned my addr 172.21.15.28:0/2573381148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926000640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280cde40 0x7fc9280ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc927400640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc92815cbb0 0x7fc92815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 -- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc92815cbb0 msgr2=0x7fc92815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc92815cbb0 0x7fc92815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 -- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280cde40 msgr2=0x7fc9280ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280cde40 0x7fc9280ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 -- 172.21.15.28:0/2573381148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280ada50 con 0x7fc928071a80 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc926a00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc9180ad040 tx=0x7fc91801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180bf410 con 0x7fc928071a80 2024-03-21T08:41:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9180bf5d0 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9280add30 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9180bf9c0 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280ae2f0 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc928073430 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.309+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc9180bd020 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.313+0000 7fc91ea00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 0x7fc90806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.313+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fc9181208d0 con 0x7fc928071a80 2024-03-21T08:41:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.313+0000 7fc926000640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 0x7fc90806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.313+0000 7fc926000640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 0x7fc90806a010 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc9280740f0 tx=0x7fc914007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.313+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc918016090 con 0x7fc928071a80 2024-03-21T08:41:51.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.477+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc92806b570 con 0x7fc928071a80 2024-03-21T08:41:51.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.477+0000 7fc91ea00640 1 -- 172.21.15.28:0/2573381148 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc9180f0910 con 0x7fc928071a80 2024-03-21T08:41:51.483 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 msgr2=0x7fc90806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 0x7fc90806a010 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc9280740f0 tx=0x7fc914007d60 comp rx=0 tx=0).stop 2024-03-21T08:41:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 msgr2=0x7fc9280af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc9180ad040 tx=0x7fc91801ac50 comp rx=0 tx=0).stop 2024-03-21T08:41:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 shutdown_connections 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc908067b50 0x7fc90806a010 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc92815cbb0 0x7fc92815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9280cde40 0x7fc9280ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 --2- 172.21.15.28:0/2573381148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc928071a80 0x7fc9280af4e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 >> 172.21.15.28:0/2573381148 conn(0x7fc9280674f0 msgr2=0x7fc928072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 shutdown_connections 2024-03-21T08:41:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.481+0000 7fc92dc00640 1 -- 172.21.15.28:0/2573381148 wait complete. 2024-03-21T08:41:51.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:51.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 Processor -- start 2024-03-21T08:41:51.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 -- start start 2024-03-21T08:41:51.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee700b0ae0 0x7fee700cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7006c490 0x7fee70099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 0x7fee7015d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700d0150 con 0x7fee7006c490 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700d0f90 con 0x7fee70099f70 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee76000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700d0380 con 0x7fee700b0ae0 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7006c490 0x7fee70099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7006c490 0x7fee70099a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36694/0 (socket says 172.21.15.28:36694) 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee67e00640 1 -- 172.21.15.28:0/3134571965 learned_addr learned my addr 172.21.15.28:0/3134571965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee74c00640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee700b0ae0 0x7fee700cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee75600640 1 --2- 172.21.15.28:0/3134571965 >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 0x7fee7015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee67e00640 1 -- 172.21.15.28:0/3134571965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee700b0ae0 msgr2=0x7fee700cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.641+0000 7fee67e00640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee700b0ae0 0x7fee700cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee67e00640 1 -- 172.21.15.28:0/3134571965 >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 msgr2=0x7fee7015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee67e00640 1 --2- 172.21.15.28:0/3134571965 >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 0x7fee7015d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee67e00640 1 -- 172.21.15.28:0/3134571965 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700d0660 con 0x7fee7006c490 2024-03-21T08:41:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee75600640 1 --2- 172.21.15.28:0/3134571965 >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 0x7fee7015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:51.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee67e00640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7006c490 0x7fee70099a30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee700738d0 tx=0x7fee5801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2515ec82f05e1035 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee6fe00640 1 -- 172.21.15.28:0/3134571965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580b0070 con 0x7fee7006c490 2024-03-21T08:41:51.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee6fe00640 1 -- 172.21.15.28:0/3134571965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee580ab600 con 0x7fee7006c490 2024-03-21T08:41:51.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee6fe00640 1 -- 172.21.15.28:0/3134571965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580ab9d0 con 0x7fee7006c490 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 -- 172.21.15.28:0/3134571965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7006c490 msgr2=0x7fee70099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7006c490 0x7fee70099a30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee700738d0 tx=0x7fee5801c520 comp rx=0 tx=0).stop 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 -- 172.21.15.28:0/3134571965 shutdown_connections 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 --2- 172.21.15.28:0/3134571965 >> v2:172.21.15.151:3300/0 conn(0x7fee70099f70 0x7fee7015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7006c490 0x7fee70099a30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 --2- 172.21.15.28:0/3134571965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee700b0ae0 0x7fee700cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 -- 172.21.15.28:0/3134571965 >> 172.21.15.28:0/3134571965 conn(0x7fee700b3390 msgr2=0x7fee700b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 -- 172.21.15.28:0/3134571965 shutdown_connections 2024-03-21T08:41:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.645+0000 7fee76000640 1 -- 172.21.15.28:0/3134571965 wait complete. 2024-03-21T08:41:51.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 Processor -- start 2024-03-21T08:41:51.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 -- start start 2024-03-21T08:41:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 0x7fee701516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 0x7fee70160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7006d9e0 con 0x7fee700b0ae0 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7006d860 con 0x7fee70099f70 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee70161910 con 0x7fee7006c490 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 0x7fee70160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee74c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee67e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 0x7fee701516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee74c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:55364/0 (socket says 172.21.15.28:55364) 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee74c00640 1 -- 172.21.15.28:0/1726455451 learned_addr learned my addr 172.21.15.28:0/1726455451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 -- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 msgr2=0x7fee701faca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 -- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 msgr2=0x7fee701516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 0x7fee701516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 -- 172.21.15.28:0/1726455451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7015f650 con 0x7fee700b0ae0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee67e00640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 0x7fee701516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee74c00640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee75600640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 0x7fee70160410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fee6800eed0 tx=0x7fee6800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68018030 con 0x7fee700b0ae0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6800bd20 con 0x7fee700b0ae0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7015f930 con 0x7fee700b0ae0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.649+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68003be0 con 0x7fee700b0ae0 2024-03-21T08:41:51.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7015f300 con 0x7fee700b0ae0 2024-03-21T08:41:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee70075270 con 0x7fee700b0ae0 2024-03-21T08:41:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fee6801b070 con 0x7fee700b0ae0 2024-03-21T08:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee6d600640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 0x7fee4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fee68086710 con 0x7fee700b0ae0 2024-03-21T08:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee74c00640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 0x7fee4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.653+0000 7fee74c00640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 0x7fee4c069fe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee60009f70 tx=0x7fee60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.657+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fee680566b0 con 0x7fee700b0ae0 2024-03-21T08:41:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.821+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee700b2980 con 0x7fee700b0ae0 2024-03-21T08:41:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee6d600640 1 -- 172.21.15.28:0/1726455451 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7fee6805a750 con 0x7fee700b0ae0 2024-03-21T08:41:51.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:51.828 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 msgr2=0x7fee4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 0x7fee4c069fe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee60009f70 tx=0x7fee60009190 comp rx=0 tx=0).stop 2024-03-21T08:41:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 msgr2=0x7fee70160410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 0x7fee70160410 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fee6800eed0 tx=0x7fee6800b480 comp rx=0 tx=0).stop 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 shutdown_connections 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee700b0ae0 0x7fee70160410 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fee70099f70 0x7fee701516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fee4c067b20 0x7fee4c069fe0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 --2- 172.21.15.28:0/1726455451 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fee7006c490 0x7fee701faca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 >> 172.21.15.28:0/1726455451 conn(0x7fee700b3390 msgr2=0x7fee70073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 shutdown_connections 2024-03-21T08:41:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.825+0000 7fee76000640 1 -- 172.21.15.28:0/1726455451 wait complete. 2024-03-21T08:41:51.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:51.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 Processor -- start 2024-03-21T08:41:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 -- start start 2024-03-21T08:41:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38980b4800 0x7f38980ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f38980cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980cd670 con 0x7f38980b4420 2024-03-21T08:41:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389815bf90 con 0x7f38980b4800 2024-03-21T08:41:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f389d600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389815b1d0 con 0x7f3898072190 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38980b4800 0x7f38980ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38980b4800 0x7f38980ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36712/0 (socket says 172.21.15.28:36712) 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 -- 172.21.15.28:0/2906123592 learned_addr learned my addr 172.21.15.28:0/2906123592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f3896a00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f38980b4420 0x7f3898071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:58940/0 (socket says 172.21.15.28:58940) 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f3897400640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f38980cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 -- 172.21.15.28:0/2906123592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 msgr2=0x7f38980cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f38980cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 -- 172.21.15.28:0/2906123592 >> v1:172.21.15.151:6789/0 conn(0x7f38980b4420 legacy=0x7f3898071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f388e000640 1 -- 172.21.15.28:0/2906123592 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389815b4b0 con 0x7f38980b4800 2024-03-21T08:41:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.989+0000 7f3896000640 1 -- 172.21.15.28:0/2906123592 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319140146 0 0) 0x7f38980cd670 con 0x7f38980b4420 2024-03-21T08:41:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f388e000640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38980b4800 0x7f38980ccbf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f388000bb10 tx=0x7f388001ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9b606c27448595 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f3896000640 1 -- 172.21.15.28:0/2906123592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800b4070 con 0x7f38980b4800 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f3896000640 1 -- 172.21.15.28:0/2906123592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38800a7bd0 con 0x7f38980b4800 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f3896000640 1 -- 172.21.15.28:0/2906123592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800af440 con 0x7f38980b4800 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 -- 172.21.15.28:0/2906123592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38980b4800 msgr2=0x7f38980ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38980b4800 0x7f38980ccbf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f388000bb10 tx=0x7f388001ec00 comp rx=0 tx=0).stop 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 -- 172.21.15.28:0/2906123592 shutdown_connections 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f38980cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 --2- 172.21.15.28:0/2906123592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38980b4800 0x7f38980ccbf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 -- 172.21.15.28:0/2906123592 >> 172.21.15.28:0/2906123592 conn(0x7f38980674f0 msgr2=0x7f389806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 -- 172.21.15.28:0/2906123592 shutdown_connections 2024-03-21T08:41:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.993+0000 7f389d600640 1 -- 172.21.15.28:0/2906123592 wait complete. 2024-03-21T08:41:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 Processor -- start 2024-03-21T08:41:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- start start 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f3898138860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 0x7f389814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389806e430 con 0x7f389814d430 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389806e2b0 con 0x7f38980b4420 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389806e5b0 con 0x7f3898072190 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f3897400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f3896a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f3898138860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f3897400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36722/0 (socket says 172.21.15.28:36722) 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f3897400640 1 -- 172.21.15.28:0/1203743284 learned_addr learned my addr 172.21.15.28:0/1203743284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 0x7f389814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 -- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 msgr2=0x7f3898138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f3898138860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 -- 172.21.15.28:0/1203743284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 msgr2=0x7f38980ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 -- 172.21.15.28:0/1203743284 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980cb250 con 0x7f38980b4420 2024-03-21T08:41:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f3897400640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388e000640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 0x7f389814cef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f38800ae000 tx=0x7f388001d440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800bf070 con 0x7f38980b4420 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38800afc30 con 0x7f38980b4420 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38800b4040 con 0x7f38980b4420 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980cb530 con 0x7f38980b4420 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898139150 con 0x7f38980b4420 2024-03-21T08:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:51.997+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389814d840 con 0x7f38980b4420 2024-03-21T08:41:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.001+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3880004030 con 0x7f38980b4420 2024-03-21T08:41:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.001+0000 7f388f400640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 0x7f3874069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.001+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f38801206b0 con 0x7f38980b4420 2024-03-21T08:41:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.001+0000 7f3896a00640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 0x7f3874069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.001+0000 7f3896a00640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 0x7f3874069fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3888007fd0 tx=0x7f3888007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.005+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f389814d840 con 0x7f38980b4420 2024-03-21T08:41:52.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.173+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3898073430 con 0x7f38980b4420 2024-03-21T08:41:52.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.173+0000 7f388f400640 1 -- 172.21.15.28:0/1203743284 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3880018820 con 0x7f38980b4420 2024-03-21T08:41:52.180 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 msgr2=0x7f3874069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 0x7f3874069fe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3888007fd0 tx=0x7f3888007d60 comp rx=0 tx=0).stop 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 msgr2=0x7f389814cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 0x7f389814cef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f38800ae000 tx=0x7f388001d440 comp rx=0 tx=0).stop 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 shutdown_connections 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f389814d430 0x7f38980ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f38980b4420 0x7f389814cef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3874067b20 0x7f3874069fe0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 --2- 172.21.15.28:0/1203743284 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3898072190 0x7f3898138860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 >> 172.21.15.28:0/1203743284 conn(0x7f38980674f0 msgr2=0x7f3898152e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 shutdown_connections 2024-03-21T08:41:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.177+0000 7f389d600640 1 -- 172.21.15.28:0/1203743284 wait complete. 2024-03-21T08:41:52.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.341+0000 7fcd89600640 1 Processor -- start 2024-03-21T08:41:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.341+0000 7fcd89600640 1 -- start start 2024-03-21T08:41:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd89600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcd8415dcf0 0x7fcd84161800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd89600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd840ceff0 0x7fcd840d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd89600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd84162730 con 0x7fcd840b9860 2024-03-21T08:41:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd89600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8415e0d0 con 0x7fcd840ceff0 2024-03-21T08:41:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd89600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd84162170 con 0x7fcd8415dcf0 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd88c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd840ceff0 0x7fcd840d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd7be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd840b9860 0x7fcd840ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41806/0 (socket says 172.21.15.28:41806) 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd7be00640 1 -- 172.21.15.28:0/3585165999 learned_addr learned my addr 172.21.15.28:0/3585165999 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd83e00640 1 --2- 172.21.15.28:0/3585165999 >> v2:172.21.15.151:3300/0 conn(0x7fcd8415dcf0 0x7fcd84161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd83400640 1 -- 172.21.15.28:0/3585165999 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007008145 0 0) 0x7fcd84162730 con 0x7fcd840b9860 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd83400640 1 -- 172.21.15.28:0/3585165999 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd68003530 con 0x7fcd840b9860 2024-03-21T08:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd88c00640 1 -- 172.21.15.28:0/3585165999 >> v2:172.21.15.151:3300/0 conn(0x7fcd8415dcf0 msgr2=0x7fcd84161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd88c00640 1 --2- 172.21.15.28:0/3585165999 >> v2:172.21.15.151:3300/0 conn(0x7fcd8415dcf0 0x7fcd84161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd88c00640 1 -- 172.21.15.28:0/3585165999 >> v1:172.21.15.28:6789/0 conn(0x7fcd840b9860 legacy=0x7fcd840ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.345+0000 7fcd88c00640 1 -- 172.21.15.28:0/3585165999 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84162450 con 0x7fcd840ceff0 2024-03-21T08:41:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd88c00640 1 --2- 172.21.15.28:0/3585165999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840ceff0 0x7fcd840d24a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd8406d8e0 tx=0x7fcd7c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec9036182d8af92 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83400640 1 -- 172.21.15.28:0/3585165999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0b2070 con 0x7fcd840ceff0 2024-03-21T08:41:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83400640 1 -- 172.21.15.28:0/3585165999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c0a59c0 con 0x7fcd840ceff0 2024-03-21T08:41:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83400640 1 -- 172.21.15.28:0/3585165999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c0a5d90 con 0x7fcd840ceff0 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- 172.21.15.28:0/3585165999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840ceff0 msgr2=0x7fcd840d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- 172.21.15.28:0/3585165999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840ceff0 0x7fcd840d24a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd8406d8e0 tx=0x7fcd7c01c900 comp rx=0 tx=0).stop 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- 172.21.15.28:0/3585165999 shutdown_connections 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- 172.21.15.28:0/3585165999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840ceff0 0x7fcd840d24a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- 172.21.15.28:0/3585165999 >> v2:172.21.15.151:3300/0 conn(0x7fcd8415dcf0 0x7fcd84161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- 172.21.15.28:0/3585165999 >> 172.21.15.28:0/3585165999 conn(0x7fcd840d0590 msgr2=0x7fcd840d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- 172.21.15.28:0/3585165999 shutdown_connections 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- 172.21.15.28:0/3585165999 wait complete. 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 Processor -- start 2024-03-21T08:41:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- start start 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd840ceff0 0x7fcd840cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd8415dcf0 0x7fcd840cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840d31f0 con 0x7fcd840b9860 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd84161f50 con 0x7fcd8415dcf0 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd89600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840d3370 con 0x7fcd840ceff0 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36734/0 (socket says 172.21.15.28:36734) 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd83e00640 1 -- 172.21.15.28:0/3286202333 learned_addr learned my addr 172.21.15.28:0/3286202333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd7be00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd840ceff0 0x7fcd840cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.349+0000 7fcd88c00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd8415dcf0 0x7fcd840cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 -- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd840ceff0 msgr2=0x7fcd840cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd840ceff0 0x7fcd840cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 -- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd8415dcf0 msgr2=0x7fcd840cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd8415dcf0 0x7fcd840cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 -- 172.21.15.28:0/3286202333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd841516d0 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd83e00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400e870 tx=0x7fcd7400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74017070 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7400c940 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd84155200 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7400ccf0 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd841557c0 con 0x7fcd840b9860 2024-03-21T08:41:52.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8404f410 con 0x7fcd840b9860 2024-03-21T08:41:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcd7401a070 con 0x7fcd840b9860 2024-03-21T08:41:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 0x7fcd60069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.353+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fcd74086210 con 0x7fcd840b9860 2024-03-21T08:41:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.357+0000 7fcd7be00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 0x7fcd60069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.357+0000 7fcd7be00640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 0x7fcd60069fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd84157c70 tx=0x7fcd6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.365+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcd74056120 con 0x7fcd840b9860 2024-03-21T08:41:52.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd840d17b0 con 0x7fcd840b9860 2024-03-21T08:41:52.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd80c00640 1 -- 172.21.15.28:0/3286202333 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd7405a1c0 con 0x7fcd840b9860 2024-03-21T08:41:52.530 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:52.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 msgr2=0x7fcd60069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 0x7fcd60069fe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd84157c70 tx=0x7fcd6c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 msgr2=0x7fcd84151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400e870 tx=0x7fcd7400ec70 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 shutdown_connections 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd8415dcf0 0x7fcd840cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd60067b20 0x7fcd60069fe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd840ceff0 0x7fcd840cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 --2- 172.21.15.28:0/3286202333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd840b9860 0x7fcd84151190 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 >> 172.21.15.28:0/3286202333 conn(0x7fcd840d0590 msgr2=0x7fcd8415cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 shutdown_connections 2024-03-21T08:41:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.525+0000 7fcd89600640 1 -- 172.21.15.28:0/3286202333 wait complete. 2024-03-21T08:41:52.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.693+0000 7f7a90200640 1 Processor -- start 2024-03-21T08:41:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.693+0000 7f7a90200640 1 -- start start 2024-03-21T08:41:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a90200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a90200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8815bf90 con 0x7f7a880b4800 2024-03-21T08:41:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a90200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8815b1d0 con 0x7f7a880b4420 2024-03-21T08:41:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a90200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8815b4b0 con 0x7f7a88072190 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8ce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7a880b4800 0x7f7a880ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41820/0 (socket says 172.21.15.28:41820) 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8ce00640 1 -- 172.21.15.28:0/3799914990 learned_addr learned my addr 172.21.15.28:0/3799914990 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8e200640 1 --2- 172.21.15.28:0/3799914990 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252497574 0 0) 0x7f7a8815bf90 con 0x7f7a880b4800 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a70003530 con 0x7f7a880b4800 2024-03-21T08:41:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751243654 0 0) 0x7f7a8815b1d0 con 0x7f7a880b4420 2024-03-21T08:41:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8e200640 1 -- 172.21.15.28:0/3799914990 >> v1:172.21.15.151:6789/0 conn(0x7f7a880b4420 legacy=0x7f7a88071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8e200640 1 -- 172.21.15.28:0/3799914990 >> v1:172.21.15.28:6789/0 conn(0x7f7a880b4800 legacy=0x7f7a880ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.697+0000 7f7a8e200640 1 -- 172.21.15.28:0/3799914990 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8815b790 con 0x7f7a88072190 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a8e200640 1 --2- 172.21.15.28:0/3799914990 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880cd300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009dc0 tx=0x7f7a8001c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a90f3a45abbb78 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a800b2070 con 0x7f7a88072190 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a800a59c0 con 0x7f7a88072190 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a87e00640 1 -- 172.21.15.28:0/3799914990 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a800a5d90 con 0x7f7a88072190 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- 172.21.15.28:0/3799914990 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 msgr2=0x7f7a880cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 --2- 172.21.15.28:0/3799914990 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880cd300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009dc0 tx=0x7f7a8001c960 comp rx=0 tx=0).stop 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- 172.21.15.28:0/3799914990 shutdown_connections 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 --2- 172.21.15.28:0/3799914990 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880cd300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- 172.21.15.28:0/3799914990 >> 172.21.15.28:0/3799914990 conn(0x7f7a880674f0 msgr2=0x7f7a8806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- 172.21.15.28:0/3799914990 shutdown_connections 2024-03-21T08:41:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- 172.21.15.28:0/3799914990 wait complete. 2024-03-21T08:41:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 Processor -- start 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.701+0000 7f7a90200640 1 -- start start 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 0x7f7a8814d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88168140 con 0x7f7a880b4420 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88167fc0 con 0x7f7a8814d400 2024-03-21T08:41:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a881682c0 con 0x7f7a88072190 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8d800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36740/0 (socket says 172.21.15.28:36740) 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 -- 172.21.15.28:0/2360331138 learned_addr learned my addr 172.21.15.28:0/2360331138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8e200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 0x7f7a8814d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 -- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 msgr2=0x7f7a880c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 -- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 msgr2=0x7f7a8814d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 0x7f7a8814d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 -- 172.21.15.28:0/2360331138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880c4c50 con 0x7f7a880b4420 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8e200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 0x7f7a8814d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a8ce00640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400c3c0 tx=0x7f7a7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74017070 con 0x7f7a880b4420 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a74005d10 con 0x7f7a880b4420 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a880c4f30 con 0x7f7a880b4420 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74007310 con 0x7f7a880b4420 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a880c54f0 con 0x7f7a880b4420 2024-03-21T08:41:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88073430 con 0x7f7a880b4420 2024-03-21T08:41:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.705+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7a7401a070 con 0x7f7a880b4420 2024-03-21T08:41:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.709+0000 7f7a85600640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 0x7f7a68069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.709+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f7a740857a0 con 0x7f7a880b4420 2024-03-21T08:41:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.709+0000 7f7a8d800640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 0x7f7a68069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.709+0000 7f7a8d800640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 0x7f7a68069fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c000f70 tx=0x7f7a7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:52.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.713+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7a740556e0 con 0x7f7a880b4420 2024-03-21T08:41:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.873+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a8804f4f0 con 0x7f7a880b4420 2024-03-21T08:41:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a85600640 1 -- 172.21.15.28:0/2360331138 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a74059780 con 0x7f7a880b4420 2024-03-21T08:41:52.876 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 msgr2=0x7f7a68069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 0x7f7a68069fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c000f70 tx=0x7f7a7c009040 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 msgr2=0x7f7a8813a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400c3c0 tx=0x7f7a7400c880 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 shutdown_connections 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a8814d400 0x7f7a8814d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a880b4420 0x7f7a8813a170 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a68067b20 0x7f7a68069fe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 --2- 172.21.15.28:0/2360331138 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a88072190 0x7f7a880c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 >> 172.21.15.28:0/2360331138 conn(0x7f7a880674f0 msgr2=0x7f7a880cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 shutdown_connections 2024-03-21T08:41:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:52.877+0000 7f7a90200640 1 -- 172.21.15.28:0/2360331138 wait complete. 2024-03-21T08:41:52.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.041+0000 7fd52f200640 1 Processor -- start 2024-03-21T08:41:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.041+0000 7fd52f200640 1 -- start start 2024-03-21T08:41:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 0x7fd5300cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5300cff70 0x7fd5300cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300ce3d0 con 0x7fd53015dba0 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300cc340 con 0x7fd5300cff70 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300d0400 con 0x7fd5300cddf0 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd53015dba0 0x7fd530071c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41828/0 (socket says 172.21.15.28:41828) 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52de00640 1 -- 172.21.15.28:0/1575536302 learned_addr learned my addr 172.21.15.28:0/1575536302 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd525400640 1 --2- 172.21.15.28:0/1575536302 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 0x7fd5300cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52e800640 1 --2- 172.21.15.28:0/1575536302 >> v2:172.21.15.28:3300/0 conn(0x7fd5300cff70 0x7fd5300cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52d400640 1 -- 172.21.15.28:0/1575536302 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213282616 0 0) 0x7fd5300ce3d0 con 0x7fd53015dba0 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd52d400640 1 -- 172.21.15.28:0/1575536302 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd514003530 con 0x7fd53015dba0 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd525400640 1 -- 172.21.15.28:0/1575536302 >> v2:172.21.15.28:3300/0 conn(0x7fd5300cff70 msgr2=0x7fd5300cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd525400640 1 --2- 172.21.15.28:0/1575536302 >> v2:172.21.15.28:3300/0 conn(0x7fd5300cff70 0x7fd5300cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd525400640 1 -- 172.21.15.28:0/1575536302 >> v1:172.21.15.28:6789/0 conn(0x7fd53015dba0 legacy=0x7fd530071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.045+0000 7fd525400640 1 -- 172.21.15.28:0/1575536302 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300ce6b0 con 0x7fd5300cddf0 2024-03-21T08:41:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd525400640 1 --2- 172.21.15.28:0/1575536302 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 0x7fd5300cfa30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd5180097c0 tx=0x7fd51801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d3d0ad2d02fcbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52d400640 1 -- 172.21.15.28:0/1575536302 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180b2070 con 0x7fd5300cddf0 2024-03-21T08:41:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52d400640 1 -- 172.21.15.28:0/1575536302 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5180ad640 con 0x7fd5300cddf0 2024-03-21T08:41:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52d400640 1 -- 172.21.15.28:0/1575536302 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180ada10 con 0x7fd5300cddf0 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- 172.21.15.28:0/1575536302 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 msgr2=0x7fd5300cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 --2- 172.21.15.28:0/1575536302 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 0x7fd5300cfa30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd5180097c0 tx=0x7fd51801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- 172.21.15.28:0/1575536302 shutdown_connections 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 --2- 172.21.15.28:0/1575536302 >> v2:172.21.15.28:3300/0 conn(0x7fd5300cff70 0x7fd5300cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 --2- 172.21.15.28:0/1575536302 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cddf0 0x7fd5300cfa30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- 172.21.15.28:0/1575536302 >> 172.21.15.28:0/1575536302 conn(0x7fd53006fa90 msgr2=0x7fd53006fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- 172.21.15.28:0/1575536302 shutdown_connections 2024-03-21T08:41:53.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- 172.21.15.28:0/1575536302 wait complete. 2024-03-21T08:41:53.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 Processor -- start 2024-03-21T08:41:53.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.049+0000 7fd52f200640 1 -- start start 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cff70 0x7fd5300b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd53015dba0 0x7fd5301538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300cd520 con 0x7fd5300cddf0 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53015e190 con 0x7fd53015dba0 2024-03-21T08:41:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300cd6a0 con 0x7fd5300cff70 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36760/0 (socket says 172.21.15.28:36760) 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 -- 172.21.15.28:0/1125725251 learned_addr learned my addr 172.21.15.28:0/1125725251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd525400640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cff70 0x7fd5300b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52e800640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd53015dba0 0x7fd5301538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 -- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cff70 msgr2=0x7fd5300b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cff70 0x7fd5300b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 -- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd53015dba0 msgr2=0x7fd5301538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd53015dba0 0x7fd5301538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 -- 172.21.15.28:0/1125725251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530155140 con 0x7fd5300cddf0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52de00640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd52000c3c0 tx=0x7fd52000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520017070 con 0x7fd5300cddf0 2024-03-21T08:41:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd520005cf0 con 0x7fd5300cddf0 2024-03-21T08:41:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5300b1600 con 0x7fd5300cddf0 2024-03-21T08:41:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5200072f0 con 0x7fd5300cddf0 2024-03-21T08:41:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300b1bc0 con 0x7fd5300cddf0 2024-03-21T08:41:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53004f410 con 0x7fd5300cddf0 2024-03-21T08:41:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.053+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd52001a070 con 0x7fd5300cddf0 2024-03-21T08:41:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.057+0000 7fd526a00640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 0x7fd50c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.057+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fd5200857c0 con 0x7fd5300cddf0 2024-03-21T08:41:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.057+0000 7fd525400640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 0x7fd50c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.057+0000 7fd525400640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 0x7fd50c069fe0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5180af000 tx=0x7fd518008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.061+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd520055640 con 0x7fd5300cddf0 2024-03-21T08:41:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.225+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd5300b7890 con 0x7fd5300cddf0 2024-03-21T08:41:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.225+0000 7fd526a00640 1 -- 172.21.15.28:0/1125725251 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd5200596e0 con 0x7fd5300cddf0 2024-03-21T08:41:53.226 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 msgr2=0x7fd50c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 0x7fd50c069fe0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5180af000 tx=0x7fd518008040 comp rx=0 tx=0).stop 2024-03-21T08:41:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 msgr2=0x7fd5301530d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd52000c3c0 tx=0x7fd52000c880 comp rx=0 tx=0).stop 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 shutdown_connections 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd53015dba0 0x7fd5301538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd50c067b20 0x7fd50c069fe0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd5300cff70 0x7fd5300b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 --2- 172.21.15.28:0/1125725251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5300cddf0 0x7fd5301530d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 >> 172.21.15.28:0/1125725251 conn(0x7fd53006fa90 msgr2=0x7fd53015e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 shutdown_connections 2024-03-21T08:41:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.229+0000 7fd52f200640 1 -- 172.21.15.28:0/1125725251 wait complete. 2024-03-21T08:41:53.276 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-21T08:41:53.276 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:41:53.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 Processor -- start 2024-03-21T08:41:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 -- start start 2024-03-21T08:41:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c15dbb0 0x7f011c072830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c072d70 0x7f011c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c0ca4c0 con 0x7f011c06de80 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0ca270 con 0x7f011c15dbb0 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0124000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0cadf0 con 0x7f011c072d70 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0121600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c15dbb0 0x7f011c072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0120c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c072d70 0x7f011c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0121600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c15dbb0 0x7f011c072830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36774/0 (socket says 172.21.15.28:36774) 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.417+0000 7f0121600640 1 -- 172.21.15.28:0/3594630621 learned_addr learned my addr 172.21.15.28:0/3594630621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:53.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0121600640 1 -- 172.21.15.28:0/3594630621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c072d70 msgr2=0x7f011c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:53.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0121600640 1 --2- 172.21.15.28:0/3594630621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c072d70 0x7f011c0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0121600640 1 -- 172.21.15.28:0/3594630621 >> v1:172.21.15.151:6789/0 conn(0x7f011c06de80 legacy=0x7f011c0cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0121600640 1 -- 172.21.15.28:0/3594630621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0ca7a0 con 0x7f011c15dbb0 2024-03-21T08:41:53.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0121600640 1 --2- 172.21.15.28:0/3594630621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c15dbb0 0x7f011c072830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f011c0ca3f0 tx=0x7f011001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bba78dcafc0c773 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f011be00640 1 -- 172.21.15.28:0/3594630621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100b0070 con 0x7f011c15dbb0 2024-03-21T08:41:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f011be00640 1 -- 172.21.15.28:0/3594630621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100ab600 con 0x7f011c15dbb0 2024-03-21T08:41:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f011be00640 1 -- 172.21.15.28:0/3594630621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100ab9d0 con 0x7f011c15dbb0 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0124000640 1 -- 172.21.15.28:0/3594630621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c15dbb0 msgr2=0x7f011c072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.421+0000 7f0124000640 1 --2- 172.21.15.28:0/3594630621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c15dbb0 0x7f011c072830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f011c0ca3f0 tx=0x7f011001c3f0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/3594630621 shutdown_connections 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 --2- 172.21.15.28:0/3594630621 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c072d70 0x7f011c0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 --2- 172.21.15.28:0/3594630621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c15dbb0 0x7f011c072830 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/3594630621 >> 172.21.15.28:0/3594630621 conn(0x7f011c0cfc20 msgr2=0x7f011c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/3594630621 shutdown_connections 2024-03-21T08:41:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/3594630621 wait complete. 2024-03-21T08:41:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 Processor -- start 2024-03-21T08:41:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- start start 2024-03-21T08:41:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c06de80 0x7f011c1f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c15dbb0 0x7f011c1fa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c16adb0 con 0x7f011c072d70 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c16ac30 con 0x7f011c15dbb0 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c16af30 con 0x7f011c06de80 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36778/0 (socket says 172.21.15.28:36778) 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 -- 172.21.15.28:0/566076943 learned_addr learned my addr 172.21.15.28:0/566076943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0122000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c15dbb0 0x7f011c1fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0121600640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c06de80 0x7f011c1f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 -- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c06de80 msgr2=0x7f011c1f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c06de80 0x7f011c1f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 -- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c15dbb0 msgr2=0x7f011c1fa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c15dbb0 0x7f011c1fa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 -- 172.21.15.28:0/566076943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0ae300 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0120c00640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f010800e900 tx=0x7f010800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108017070 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010800ca00 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800cdf0 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c0ada60 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.425+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c0ae020 con 0x7f011c072d70 2024-03-21T08:41:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c04f410 con 0x7f011c072d70 2024-03-21T08:41:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f010801a070 con 0x7f011c072d70 2024-03-21T08:41:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0119600640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 0x7f00fc06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f0108086670 con 0x7f011c072d70 2024-03-21T08:41:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0121600640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 0x7f00fc06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.429+0000 7f0121600640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 0x7f00fc06a010 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f01100ad000 tx=0x7f0110008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.433+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0108056580 con 0x7f011c072d70 2024-03-21T08:41:53.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f011c15d3e0 con 0x7f011c072d70 2024-03-21T08:41:53.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0119600640 1 -- 172.21.15.28:0/566076943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f010805a620 con 0x7f011c072d70 2024-03-21T08:41:53.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:53.601 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:53.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 msgr2=0x7f00fc06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 0x7f00fc06a010 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f01100ad000 tx=0x7f0110008040 comp rx=0 tx=0).stop 2024-03-21T08:41:53.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 msgr2=0x7f011c15e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f010800e900 tx=0x7f010800edc0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 shutdown_connections 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f011c15dbb0 0x7f011c1fa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c072d70 0x7f011c15e360 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b50 0x7f00fc06a010 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 --2- 172.21.15.28:0/566076943 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f011c06de80 0x7f011c1f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 >> 172.21.15.28:0/566076943 conn(0x7f011c0cfc20 msgr2=0x7f011c0b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 shutdown_connections 2024-03-21T08:41:53.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.601+0000 7f0124000640 1 -- 172.21.15.28:0/566076943 wait complete. 2024-03-21T08:41:53.623 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:53.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.765+0000 7f8916800640 1 Processor -- start 2024-03-21T08:41:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.765+0000 7f8916800640 1 -- start start 2024-03-21T08:41:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8916800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f891006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8916800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89100ce5b0 con 0x7f89100cde40 2024-03-21T08:41:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8916800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89100ce890 con 0x7f891015cbb0 2024-03-21T08:41:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8916800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891015c720 con 0x7f8910071a80 2024-03-21T08:41:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8906a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89100cde40 0x7f8910071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41844/0 (socket says 172.21.15.28:41844) 2024-03-21T08:41:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f8906a00640 1 -- 172.21.15.28:0/823916424 learned_addr learned my addr 172.21.15.28:0/823916424 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890f400640 1 --2- 172.21.15.28:0/823916424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f891006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152089158 0 0) 0x7f89100ce5b0 con 0x7f89100cde40 2024-03-21T08:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f4003530 con 0x7f89100cde40 2024-03-21T08:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580797155 0 0) 0x7f89100ce890 con 0x7f891015cbb0 2024-03-21T08:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89100ce5b0 con 0x7f891015cbb0 2024-03-21T08:41:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890f400640 1 -- 172.21.15.28:0/823916424 >> v1:172.21.15.151:6789/0 conn(0x7f891015cbb0 legacy=0x7f89100cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890f400640 1 -- 172.21.15.28:0/823916424 >> v1:172.21.15.28:6789/0 conn(0x7f89100cde40 legacy=0x7f8910071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.769+0000 7f890f400640 1 -- 172.21.15.28:0/823916424 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891015c3b0 con 0x7f8910071a80 2024-03-21T08:41:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f890f400640 1 --2- 172.21.15.28:0/823916424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f891006f0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f89000097f0 tx=0x7f890001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fd31b7ffa1937c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000a7070 con 0x7f8910071a80 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89000b0070 con 0x7f8910071a80 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f890ea00640 1 -- 172.21.15.28:0/823916424 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000ab890 con 0x7f8910071a80 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- 172.21.15.28:0/823916424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 msgr2=0x7f891006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 --2- 172.21.15.28:0/823916424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f891006f0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f89000097f0 tx=0x7f890001c450 comp rx=0 tx=0).stop 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- 172.21.15.28:0/823916424 shutdown_connections 2024-03-21T08:41:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 --2- 172.21.15.28:0/823916424 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f891006f0a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- 172.21.15.28:0/823916424 >> 172.21.15.28:0/823916424 conn(0x7f89100674f0 msgr2=0x7f891006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- 172.21.15.28:0/823916424 shutdown_connections 2024-03-21T08:41:53.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- 172.21.15.28:0/823916424 wait complete. 2024-03-21T08:41:53.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 Processor -- start 2024-03-21T08:41:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.773+0000 7f8916800640 1 -- start start 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f89100b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f891015cbb0 0x7f89100b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891016a0b0 con 0x7f89100cde40 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910169f30 con 0x7f891015cbb0 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891016a230 con 0x7f8910071a80 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36784/0 (socket says 172.21.15.28:36784) 2024-03-21T08:41:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 -- 172.21.15.28:0/3236448904 learned_addr learned my addr 172.21.15.28:0/3236448904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f890fe00640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f891015cbb0 0x7f89100b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f890f400640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f89100b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 -- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 msgr2=0x7f89100b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f89100b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 -- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f891015cbb0 msgr2=0x7f89100b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f891015cbb0 0x7f89100b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 -- 172.21.15.28:0/3236448904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100ad200 con 0x7f89100cde40 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8906a00640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c500 tx=0x7f88f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8017070 con 0x7f89100cde40 2024-03-21T08:41:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8005cf0 con 0x7f89100cde40 2024-03-21T08:41:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80072f0 con 0x7f89100cde40 2024-03-21T08:41:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89100ad4e0 con 0x7f89100cde40 2024-03-21T08:41:53.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89100adaa0 con 0x7f89100cde40 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.777+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8910073430 con 0x7f89100cde40 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.781+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f88f801a070 con 0x7f89100cde40 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.781+0000 7f8907e00640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 0x7f88ec069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.781+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f88f80856d0 con 0x7f89100cde40 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.781+0000 7f890f400640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 0x7f88ec069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.781+0000 7f890f400640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 0x7f88ec069fe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89000af000 tx=0x7f89000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:53.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.785+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f88f8055670 con 0x7f89100cde40 2024-03-21T08:41:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.945+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f891004f4f0 con 0x7f89100cde40 2024-03-21T08:41:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.945+0000 7f8907e00640 1 -- 172.21.15.28:0/3236448904 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88f8059710 con 0x7f89100cde40 2024-03-21T08:41:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 msgr2=0x7f88ec069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 0x7f88ec069fe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89000af000 tx=0x7f89000a7000 comp rx=0 tx=0).stop 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 msgr2=0x7f89100ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c500 tx=0x7f88f800c9c0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 shutdown_connections 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f891015cbb0 0x7f89100b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89100cde40 0x7f89100ae2a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f88ec067b20 0x7f88ec069fe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 --2- 172.21.15.28:0/3236448904 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8910071a80 0x7f89100b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 >> 172.21.15.28:0/3236448904 conn(0x7f89100674f0 msgr2=0x7f891013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 shutdown_connections 2024-03-21T08:41:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:53.949+0000 7f8916800640 1 -- 172.21.15.28:0/3236448904 wait complete. 2024-03-21T08:41:53.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:54.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.109+0000 7fabb3400640 1 Processor -- start 2024-03-21T08:41:54.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.109+0000 7fabb3400640 1 -- start start 2024-03-21T08:41:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0b4420 0x7fabac071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fabac0cd300 0x7fabac071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb3400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac15bf90 con 0x7fabac0b4800 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb3400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac15b1d0 con 0x7fabac0cd300 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb3400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac15b4b0 con 0x7fabac0b4420 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fabac0cd300 0x7fabac071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fababe00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fabac0b4800 0x7fabac0ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:58988/0 (socket says 172.21.15.28:58988) 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fababe00640 1 -- 172.21.15.28:0/4060807214 learned_addr learned my addr 172.21.15.28:0/4060807214 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb0a00640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0b4420 0x7fabac071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb1400640 1 -- 172.21.15.28:0/4060807214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0b4420 msgr2=0x7fabac071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb1400640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0b4420 0x7fabac071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb1400640 1 -- 172.21.15.28:0/4060807214 >> v1:172.21.15.151:6789/0 conn(0x7fabac0b4800 legacy=0x7fabac0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.113+0000 7fabb1400640 1 -- 172.21.15.28:0/4060807214 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac15b790 con 0x7fabac0cd300 2024-03-21T08:41:54.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb1400640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac0cd300 0x7fabac071fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faba4009d90 tx=0x7faba401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39433a57f4807db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabab400640 1 -- 172.21.15.28:0/4060807214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40b2070 con 0x7fabac0cd300 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabab400640 1 -- 172.21.15.28:0/4060807214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba40a5900 con 0x7fabac0cd300 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabab400640 1 -- 172.21.15.28:0/4060807214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40a5cd0 con 0x7fabac0cd300 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- 172.21.15.28:0/4060807214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac0cd300 msgr2=0x7fabac071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac0cd300 0x7fabac071fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7faba4009d90 tx=0x7faba401c8b0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- 172.21.15.28:0/4060807214 shutdown_connections 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac0cd300 0x7fabac071fc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 --2- 172.21.15.28:0/4060807214 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0b4420 0x7fabac071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- 172.21.15.28:0/4060807214 >> 172.21.15.28:0/4060807214 conn(0x7fabac0674f0 msgr2=0x7fabac06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- 172.21.15.28:0/4060807214 shutdown_connections 2024-03-21T08:41:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- 172.21.15.28:0/4060807214 wait complete. 2024-03-21T08:41:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 Processor -- start 2024-03-21T08:41:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.117+0000 7fabb3400640 1 -- start start 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fabac0b4420 0x7fabac150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0cd300 0x7fabac1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0b36b0 con 0x7fabac157d20 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0b3530 con 0x7fabac0b4420 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0b3830 con 0x7fabac0cd300 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb0a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fabac0b4420 0x7fabac150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36800/0 (socket says 172.21.15.28:36800) 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 -- 172.21.15.28:0/2232458026 learned_addr learned my addr 172.21.15.28:0/2232458026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fababe00640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0cd300 0x7fabac1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 -- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0cd300 msgr2=0x7fabac1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0cd300 0x7fabac1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 -- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fabac0b4420 msgr2=0x7fabac150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fabac0b4420 0x7fabac150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 -- 172.21.15.28:0/2232458026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac156740 con 0x7fabac157d20 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb1400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faba40b0040 tx=0x7faba4019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40b2050 con 0x7fabac157d20 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba40be070 con 0x7fabac157d20 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40ada90 con 0x7fabac157d20 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac156a20 con 0x7fabac157d20 2024-03-21T08:41:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac156fe0 con 0x7fabac157d20 2024-03-21T08:41:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac068320 con 0x7fabac157d20 2024-03-21T08:41:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.121+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faba40a5780 con 0x7fabac157d20 2024-03-21T08:41:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.125+0000 7faba8c00640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 0x7fab8c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.125+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7faba4121330 con 0x7fabac157d20 2024-03-21T08:41:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.125+0000 7fabb0a00640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 0x7fab8c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.125+0000 7fabb0a00640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 0x7fab8c069fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faba0009ac0 tx=0x7faba0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.129+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faba4016560 con 0x7fabac157d20 2024-03-21T08:41:54.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fabac06b660 con 0x7fabac157d20 2024-03-21T08:41:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7faba8c00640 1 -- 172.21.15.28:0/2232458026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7faba40f13f0 con 0x7fabac157d20 2024-03-21T08:41:54.293 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 msgr2=0x7fab8c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 0x7fab8c069fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faba0009ac0 tx=0x7faba0009190 comp rx=0 tx=0).stop 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 msgr2=0x7fabac15a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faba40b0040 tx=0x7faba4019650 comp rx=0 tx=0).stop 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 shutdown_connections 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabac157d20 0x7fabac15a070 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab8c067b20 0x7fab8c069fe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fabac0cd300 0x7fabac1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 --2- 172.21.15.28:0/2232458026 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fabac0b4420 0x7fabac150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 >> 172.21.15.28:0/2232458026 conn(0x7fabac0674f0 msgr2=0x7fabac151280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 shutdown_connections 2024-03-21T08:41:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.292+0000 7fabb3400640 1 -- 172.21.15.28:0/2232458026 wait complete. 2024-03-21T08:41:54.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 Processor -- start 2024-03-21T08:41:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 -- start start 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64600b4420 0x7f6460071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64600b4800 0x7f6460071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646015bf90 con 0x7f64600ccbf0 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646015b1d0 con 0x7f64600b4800 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6467000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646015b4b0 con 0x7f64600b4420 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6457400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64600b4800 0x7f6460071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6465000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64600ccbf0 0x7f64600cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41846/0 (socket says 172.21.15.28:41846) 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f6465000640 1 -- 172.21.15.28:0/2386573221 learned_addr learned my addr 172.21.15.28:0/2386573221 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.456+0000 7f645fe00640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64600b4420 0x7f6460071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6457400640 1 -- 172.21.15.28:0/2386573221 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64600b4420 msgr2=0x7f6460071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6457400640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64600b4420 0x7f6460071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6457400640 1 -- 172.21.15.28:0/2386573221 >> v1:172.21.15.28:6789/0 conn(0x7f64600ccbf0 legacy=0x7f64600cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6457400640 1 -- 172.21.15.28:0/2386573221 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646015b790 con 0x7f64600b4800 2024-03-21T08:41:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6457400640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600b4800 0x7f6460071fc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f64480097c0 tx=0x7f644801e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843bad4a42ef4892 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f645f400640 1 -- 172.21.15.28:0/2386573221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64480ac5f0 con 0x7f64600b4800 2024-03-21T08:41:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f645f400640 1 -- 172.21.15.28:0/2386573221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64480b1070 con 0x7f64600b4800 2024-03-21T08:41:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f645f400640 1 -- 172.21.15.28:0/2386573221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64480ac9b0 con 0x7f64600b4800 2024-03-21T08:41:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6467000640 1 -- 172.21.15.28:0/2386573221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600b4800 msgr2=0x7f6460071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6467000640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600b4800 0x7f6460071fc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f64480097c0 tx=0x7f644801e5f0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.460+0000 7f6467000640 1 -- 172.21.15.28:0/2386573221 shutdown_connections 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600b4800 0x7f6460071fc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 --2- 172.21.15.28:0/2386573221 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64600b4420 0x7f6460071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- 172.21.15.28:0/2386573221 >> 172.21.15.28:0/2386573221 conn(0x7f64600674f0 msgr2=0x7f646006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- 172.21.15.28:0/2386573221 shutdown_connections 2024-03-21T08:41:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- 172.21.15.28:0/2386573221 wait complete. 2024-03-21T08:41:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 Processor -- start 2024-03-21T08:41:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- start start 2024-03-21T08:41:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 0x7f6460148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f646014de70 0x7f64601457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646006e6c0 con 0x7f64600ccbf0 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646006e540 con 0x7f64600b4420 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6467000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646006e840 con 0x7f646014de70 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6457400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6457400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36826/0 (socket says 172.21.15.28:36826) 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6457400640 1 -- 172.21.15.28:0/2481687411 learned_addr learned my addr 172.21.15.28:0/2481687411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 0x7f6460148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6465000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f646014de70 0x7f64601457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 -- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f646014de70 msgr2=0x7f64601457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f646014de70 0x7f64601457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 -- 172.21.15.28:0/2481687411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 msgr2=0x7f646014d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 -- 172.21.15.28:0/2481687411 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64601485a0 con 0x7f64600b4420 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f6457400640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.464+0000 7f645fe00640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 0x7f6460148060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f645000e8d0 tx=0x7f645000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450017070 con 0x7f64600b4420 2024-03-21T08:41:54.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645000c940 con 0x7f64600b4420 2024-03-21T08:41:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000cd10 con 0x7f64600b4420 2024-03-21T08:41:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6460146460 con 0x7f64600b4420 2024-03-21T08:41:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6460146a20 con 0x7f64600b4420 2024-03-21T08:41:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646014e280 con 0x7f64600b4420 2024-03-21T08:41:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f645001a070 con 0x7f64600b4420 2024-03-21T08:41:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 0x7f643c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f6450085fc0 con 0x7f64600b4420 2024-03-21T08:41:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.468+0000 7f6457400640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 0x7f643c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.472+0000 7f6457400640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 0x7f643c06a010 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64480097c0 tx=0x7f64480a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.472+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f646014e280 con 0x7f64600b4420 2024-03-21T08:41:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.636+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6460073430 con 0x7f64600b4420 2024-03-21T08:41:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.636+0000 7f645cc00640 1 -- 172.21.15.28:0/2481687411 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6450055fe0 con 0x7f64600b4420 2024-03-21T08:41:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 msgr2=0x7f643c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 0x7f643c06a010 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64480097c0 tx=0x7f64480a9000 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 msgr2=0x7f6460148060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 0x7f6460148060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f645000e8d0 tx=0x7f645000ed90 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 shutdown_connections 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f643c067b50 0x7f643c06a010 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f646014de70 0x7f64601457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64600ccbf0 0x7f646014d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 --2- 172.21.15.28:0/2481687411 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64600b4420 0x7f6460148060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 >> 172.21.15.28:0/2481687411 conn(0x7f64600674f0 msgr2=0x7f6460159ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 shutdown_connections 2024-03-21T08:41:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.640+0000 7f6467000640 1 -- 172.21.15.28:0/2481687411 wait complete. 2024-03-21T08:41:54.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 Processor -- start 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 -- start start 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0806f0a0 0x7fbc080cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc0815cbb0 0x7fbc080cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0806f500 con 0x7fbc08071a80 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc080ce380 con 0x7fbc0815cbb0 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0815c720 con 0x7fbc0806f0a0 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc0815cbb0 0x7fbc080cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc0815cbb0 0x7fbc080cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 -- 172.21.15.28:0/3947066817 learned_addr learned my addr 172.21.15.28:0/3947066817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc07400640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0806f0a0 0x7fbc080cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc07e00640 1 --1- 172.21.15.28:0/3947066817 >> v1:172.21.15.151:6789/0 conn(0x7fbc08071a80 0x7fbc08071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:58992/0 (socket says 172.21.15.28:58992) 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 -- 172.21.15.28:0/3947066817 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0806f0a0 msgr2=0x7fbc080cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0806f0a0 0x7fbc080cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 -- 172.21.15.28:0/3947066817 >> v1:172.21.15.151:6789/0 conn(0x7fbc08071a80 legacy=0x7fbc08071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.804+0000 7fbc0ce00640 1 -- 172.21.15.28:0/3947066817 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080ce5b0 con 0x7fbc0815cbb0 2024-03-21T08:41:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ce00640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0815cbb0 0x7fbc080cc4b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009e50 tx=0x7fbc0001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cd838481b58d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc06a00640 1 -- 172.21.15.28:0/3947066817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000a5830 con 0x7fbc0815cbb0 2024-03-21T08:41:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc06a00640 1 -- 172.21.15.28:0/3947066817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc000a59d0 con 0x7fbc0815cbb0 2024-03-21T08:41:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc06a00640 1 -- 172.21.15.28:0/3947066817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000a5da0 con 0x7fbc0815cbb0 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3947066817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0815cbb0 msgr2=0x7fbc080cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0815cbb0 0x7fbc080cc4b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009e50 tx=0x7fbc0001c970 comp rx=0 tx=0).stop 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3947066817 shutdown_connections 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0815cbb0 0x7fbc080cc4b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3947066817 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0806f0a0 0x7fbc080cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3947066817 >> 172.21.15.28:0/3947066817 conn(0x7fbc080674f0 msgr2=0x7fbc0806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3947066817 shutdown_connections 2024-03-21T08:41:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3947066817 wait complete. 2024-03-21T08:41:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 Processor -- start 2024-03-21T08:41:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.808+0000 7fbc0ee00640 1 -- start start 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc08071a80 0x7fbc0813bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0815cbb0 0x7fbc080b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08144b20 con 0x7fbc0806f0a0 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080cce00 con 0x7fbc08071a80 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08168f20 con 0x7fbc0815cbb0 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36854/0 (socket says 172.21.15.28:36854) 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 -- 172.21.15.28:0/3958381744 learned_addr learned my addr 172.21.15.28:0/3958381744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ce00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0815cbb0 0x7fbc080b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07400640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc08071a80 0x7fbc0813bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 -- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0815cbb0 msgr2=0x7fbc080b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0815cbb0 0x7fbc080b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 -- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc08071a80 msgr2=0x7fbc0813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc08071a80 0x7fbc0813bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 -- 172.21.15.28:0/3958381744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08144330 con 0x7fbc0806f0a0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc07e00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c450 tx=0x7fbbf800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8018070 con 0x7fbc0806f0a0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf8007480 con 0x7fbc0806f0a0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80047f0 con 0x7fbc0806f0a0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc08143290 con 0x7fbc0806f0a0 2024-03-21T08:41:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc08143850 con 0x7fbc0806f0a0 2024-03-21T08:41:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.812+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc08073430 con 0x7fbc0806f0a0 2024-03-21T08:41:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.816+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbbf801b070 con 0x7fbc0806f0a0 2024-03-21T08:41:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.816+0000 7fbbffe00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 0x7fbbe8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.816+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fbbf80854e0 con 0x7fbc0806f0a0 2024-03-21T08:41:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.816+0000 7fbc07400640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 0x7fbbe8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.816+0000 7fbc07400640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 0x7fbbe8069fe0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbc080aecf0 tx=0x7fbbf4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.820+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbbf80553f0 con 0x7fbc0806f0a0 2024-03-21T08:41:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.980+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbc0804f4f0 con 0x7fbc0806f0a0 2024-03-21T08:41:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbbffe00640 1 -- 172.21.15.28:0/3958381744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbbf8059490 con 0x7fbc0806f0a0 2024-03-21T08:41:54.984 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 msgr2=0x7fbbe8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 0x7fbbe8069fe0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbc080aecf0 tx=0x7fbbf4009190 comp rx=0 tx=0).stop 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 msgr2=0x7fbc081fb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c450 tx=0x7fbbf800c910 comp rx=0 tx=0).stop 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 shutdown_connections 2024-03-21T08:41:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbbe8067b20 0x7fbbe8069fe0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc0815cbb0 0x7fbc080b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc08071a80 0x7fbc0813bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 --2- 172.21.15.28:0/3958381744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0806f0a0 0x7fbc081fb260 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 >> 172.21.15.28:0/3958381744 conn(0x7fbc080674f0 msgr2=0x7fbc080b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 shutdown_connections 2024-03-21T08:41:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:54.984+0000 7fbc0ee00640 1 -- 172.21.15.28:0/3958381744 wait complete. 2024-03-21T08:41:55.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-21T08:41:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.144+0000 7f792ee00640 1 Processor -- start 2024-03-21T08:41:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.144+0000 7f792ee00640 1 -- start start 2024-03-21T08:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f792ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79280af060 0x7f79280cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f792ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792815bf00 con 0x7f79280b34f0 2024-03-21T08:41:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f792ee00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79280b3980 con 0x7f79280b5520 2024-03-21T08:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f792ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79280ce6e0 con 0x7f79280af060 2024-03-21T08:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f7927e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79280af060 0x7f79280cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f791fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f79280b34f0 0x7f79280b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41862/0 (socket says 172.21.15.28:41862) 2024-03-21T08:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f791fe00640 1 -- 172.21.15.28:0/1112387870 learned_addr learned my addr 172.21.15.28:0/1112387870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f7927e00640 1 -- 172.21.15.28:0/1112387870 >> v1:172.21.15.151:6789/0 conn(0x7f79280b5520 legacy=0x7f792815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f7927e00640 1 -- 172.21.15.28:0/1112387870 >> v1:172.21.15.28:6789/0 conn(0x7f79280b34f0 legacy=0x7f79280b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f7927e00640 1 -- 172.21.15.28:0/1112387870 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280cdad0 con 0x7f79280af060 2024-03-21T08:41:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.148+0000 7f7927400640 1 -- 172.21.15.28:0/1112387870 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255694796 0 0) 0x7f792815bf00 con 0x7f79280b34f0 2024-03-21T08:41:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f7927e00640 1 --2- 172.21.15.28:0/1112387870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280af060 0x7f79280cd380 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f79280ce860 tx=0x7f791801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8f88dade86f676 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f7927400640 1 -- 172.21.15.28:0/1112387870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180b0070 con 0x7f79280af060 2024-03-21T08:41:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f7927400640 1 -- 172.21.15.28:0/1112387870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180ab600 con 0x7f79280af060 2024-03-21T08:41:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f7927400640 1 -- 172.21.15.28:0/1112387870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180ab9d0 con 0x7f79280af060 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- 172.21.15.28:0/1112387870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280af060 msgr2=0x7f79280cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 --2- 172.21.15.28:0/1112387870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280af060 0x7f79280cd380 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f79280ce860 tx=0x7f791801c410 comp rx=0 tx=0).stop 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- 172.21.15.28:0/1112387870 shutdown_connections 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 --2- 172.21.15.28:0/1112387870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280af060 0x7f79280cd380 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- 172.21.15.28:0/1112387870 >> 172.21.15.28:0/1112387870 conn(0x7f79280cc230 msgr2=0x7f79280cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- 172.21.15.28:0/1112387870 shutdown_connections 2024-03-21T08:41:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- 172.21.15.28:0/1112387870 wait complete. 2024-03-21T08:41:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 Processor -- start 2024-03-21T08:41:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 -- start start 2024-03-21T08:41:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.152+0000 7f792ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79280b34f0 0x7f7928147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79281492e0 0x7f79280b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d2a0 con 0x7f79280b5520 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d120 con 0x7f79280b34f0 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792815d420 con 0x7f79281492e0 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f7927e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79280b34f0 0x7f7928147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36860/0 (socket says 172.21.15.28:36860) 2024-03-21T08:41:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 -- 172.21.15.28:0/3019332156 learned_addr learned my addr 172.21.15.28:0/3019332156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ce00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79281492e0 0x7f79280b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 -- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79281492e0 msgr2=0x7f79280b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79281492e0 0x7f79280b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 -- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79280b34f0 msgr2=0x7f7928147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79280b34f0 0x7f7928147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 -- 172.21.15.28:0/3019332156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7928153980 con 0x7f79280b5520 2024-03-21T08:41:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f791fe00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f791000c4a0 tx=0x7f791000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910018070 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79100074e0 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928153c60 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910004a30 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7928154220 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792804f410 con 0x7f79280b5520 2024-03-21T08:41:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.156+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f791001b070 con 0x7f79280b5520 2024-03-21T08:41:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.160+0000 7f7924c00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 0x7f7904069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.160+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f7910085640 con 0x7f79280b5520 2024-03-21T08:41:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.160+0000 7f7927e00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 0x7f7904069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.160+0000 7f7927e00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 0x7f7904069fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79180ad000 tx=0x7f79180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.164+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7910055550 con 0x7f79280b5520 2024-03-21T08:41:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:55.384+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7f79280cc820 con 0x7f79280b5520 2024-03-21T08:41:56.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.168+0000 7f7924c00640 1 -- 172.21.15.28:0/3019332156 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v68) v1 ==== 150+0+0 (secure 0 0 0) 0x7f79100595f0 con 0x7f79280b5520 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 msgr2=0x7f7904069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 0x7f7904069fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79180ad000 tx=0x7f79180a6000 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 msgr2=0x7f79281f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f791000c4a0 tx=0x7f791000c960 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 shutdown_connections 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7904067b20 0x7f7904069fe0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79281492e0 0x7f79280b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79280b5520 0x7f79281f69f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 --2- 172.21.15.28:0/3019332156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79280b34f0 0x7f7928147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 >> 172.21.15.28:0/3019332156 conn(0x7f79280cc230 msgr2=0x7f792815e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:56.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 shutdown_connections 2024-03-21T08:41:56.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.172+0000 7f792ee00640 1 -- 172.21.15.28:0/3019332156 wait complete. 2024-03-21T08:41:56.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-21T08:41:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 Processor -- start 2024-03-21T08:41:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 -- start start 2024-03-21T08:41:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7d415e2a0 0x7fa7d409aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa7d4099c90 0x7fa7d409a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d40d0930 con 0x7fa7d406c520 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d409a630 con 0x7fa7d415e2a0 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d8e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d40d0c10 con 0x7fa7d4099c90 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7d415e2a0 0x7fa7d409aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7d415e2a0 0x7fa7d409aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36870/0 (socket says 172.21.15.28:36870) 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 -- 172.21.15.28:0/1302448774 learned_addr learned my addr 172.21.15.28:0/1302448774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7cbe00640 1 --2- 172.21.15.28:0/1302448774 >> v2:172.21.15.151:3300/0 conn(0x7fa7d4099c90 0x7fa7d409a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 -- 172.21.15.28:0/1302448774 >> v2:172.21.15.151:3300/0 conn(0x7fa7d4099c90 msgr2=0x7fa7d409a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 --2- 172.21.15.28:0/1302448774 >> v2:172.21.15.151:3300/0 conn(0x7fa7d4099c90 0x7fa7d409a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 -- 172.21.15.28:0/1302448774 >> v1:172.21.15.151:6789/0 conn(0x7fa7d406c520 legacy=0x7fa7d406c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.348+0000 7fa7d1e00640 1 -- 172.21.15.28:0/1302448774 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40d0ef0 con 0x7fa7d415e2a0 2024-03-21T08:41:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d1e00640 1 --2- 172.21.15.28:0/1302448774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d415e2a0 0x7fa7d409aad0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4072bb0 tx=0x7fa7c401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d45404445f4c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d1400640 1 -- 172.21.15.28:0/1302448774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40b1070 con 0x7fa7d415e2a0 2024-03-21T08:41:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d1400640 1 -- 172.21.15.28:0/1302448774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c40ac600 con 0x7fa7d415e2a0 2024-03-21T08:41:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d1400640 1 -- 172.21.15.28:0/1302448774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40ac9d0 con 0x7fa7d415e2a0 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- 172.21.15.28:0/1302448774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d415e2a0 msgr2=0x7fa7d409aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/1302448774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d415e2a0 0x7fa7d409aad0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4072bb0 tx=0x7fa7c401c490 comp rx=0 tx=0).stop 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- 172.21.15.28:0/1302448774 shutdown_connections 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/1302448774 >> v2:172.21.15.151:3300/0 conn(0x7fa7d4099c90 0x7fa7d409a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/1302448774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d415e2a0 0x7fa7d409aad0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- 172.21.15.28:0/1302448774 >> 172.21.15.28:0/1302448774 conn(0x7fa7d40b3620 msgr2=0x7fa7d40b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- 172.21.15.28:0/1302448774 shutdown_connections 2024-03-21T08:41:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- 172.21.15.28:0/1302448774 wait complete. 2024-03-21T08:41:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 Processor -- start 2024-03-21T08:41:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.352+0000 7fa7d8e00640 1 -- start start 2024-03-21T08:41:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 0x7fa7d415f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7d415e2a0 0x7fa7d415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d409b190 con 0x7fa7d406c520 2024-03-21T08:41:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d409b010 con 0x7fa7d415e2a0 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d416b6f0 con 0x7fa7d4099c90 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 0x7fa7d415f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36886/0 (socket says 172.21.15.28:36886) 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 -- 172.21.15.28:0/2023620029 learned_addr learned my addr 172.21.15.28:0/2023620029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d2800640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7d415e2a0 0x7fa7d415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 -- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 msgr2=0x7fa7d415f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 0x7fa7d415f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 -- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7d415e2a0 msgr2=0x7fa7d415c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7d415e2a0 0x7fa7d415c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 -- 172.21.15.28:0/2023620029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40cbb70 con 0x7fa7d406c520 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7cbe00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 0x7fa7d415f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d1e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40190a0 tx=0x7fa7c401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40b1040 con 0x7fa7d406c520 2024-03-21T08:41:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c40bc070 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d415fbf0 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40b7650 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d41601b0 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d409faf0 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.356+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa7c40b77f0 con 0x7fa7d406c520 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.360+0000 7fa7ca000640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 0x7fa7b0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.360+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fa7c4120420 con 0x7fa7d406c520 2024-03-21T08:41:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.360+0000 7fa7cbe00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 0x7fa7b0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.360+0000 7fa7cbe00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 0x7fa7b0069fe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4160890 tx=0x7fa7bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.364+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa7d40cbb70 con 0x7fa7d406c520 2024-03-21T08:41:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.528+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa7d415ac40 con 0x7fa7d406c520 2024-03-21T08:41:56.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.528+0000 7fa7ca000640 1 -- 172.21.15.28:0/2023620029 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7c4016110 con 0x7fa7d406c520 2024-03-21T08:41:56.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 msgr2=0x7fa7b0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 0x7fa7b0069fe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4160890 tx=0x7fa7bc009190 comp rx=0 tx=0).stop 2024-03-21T08:41:56.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 msgr2=0x7fa7d40cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40190a0 tx=0x7fa7c401acd0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 shutdown_connections 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa7d415e2a0 0x7fa7d415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa7b0067b20 0x7fa7b0069fe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa7d4099c90 0x7fa7d415f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 --2- 172.21.15.28:0/2023620029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7d406c520 0x7fa7d40cb630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 >> 172.21.15.28:0/2023620029 conn(0x7fa7d40b3620 msgr2=0x7fa7d406d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 shutdown_connections 2024-03-21T08:41:56.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:56.532+0000 7fa7d8e00640 1 -- 172.21.15.28:0/2023620029 wait complete. 2024-03-21T08:41:56.553 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-21T08:41:56.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:41:56.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:41:56.745 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-21T08:41:56.745 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:41:56.745 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:41:56.754 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:41:56.754 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:41:56.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.060+0000 7f2ea7e00640 1 Processor -- start 2024-03-21T08:41:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.060+0000 7f2ea7e00640 1 -- start start 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2ea80cd3a0 0x7f2ea815c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea806b6e0 con 0x7f2ea80cdbf0 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea806b9c0 con 0x7f2ea806b170 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea8070220 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2ea80cdbf0 0x7f2ea80cdfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41866/0 (socket says 172.21.15.28:41866) 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6a00640 1 -- 172.21.15.28:0/2812802656 learned_addr learned my addr 172.21.15.28:0/2812802656 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2812802656 >> v2:172.21.15.151:3300/0 conn(0x7f2ea80cd3a0 0x7f2ea815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830450955 0 0) 0x7f2ea806b6e0 con 0x7f2ea80cdbf0 2024-03-21T08:41:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c003530 con 0x7f2ea80cdbf0 2024-03-21T08:41:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385047956 0 0) 0x7f2ea806b9c0 con 0x7f2ea806b170 2024-03-21T08:41:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea806b6e0 con 0x7f2ea806b170 2024-03-21T08:41:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7400640 1 -- 172.21.15.28:0/2812802656 >> v1:172.21.15.151:6789/0 conn(0x7f2ea806b170 legacy=0x7f2ea8070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7400640 1 -- 172.21.15.28:0/2812802656 >> v1:172.21.15.28:6789/0 conn(0x7f2ea80cdbf0 legacy=0x7f2ea80cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7400640 1 -- 172.21.15.28:0/2812802656 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8070500 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.064+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2812802656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea815c250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b2c9ef4d03ec65a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00b0070 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00ab600 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6000640 1 -- 172.21.15.28:0/2812802656 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00ab9b0 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2812802656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 msgr2=0x7f2ea815c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2812802656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea815c250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea001c3e0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2812802656 shutdown_connections 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2812802656 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea815c250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2812802656 >> 172.21.15.28:0/2812802656 conn(0x7f2ea806e920 msgr2=0x7f2ea806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2812802656 shutdown_connections 2024-03-21T08:41:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2812802656 wait complete. 2024-03-21T08:41:57.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 Processor -- start 2024-03-21T08:41:57.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- start start 2024-03-21T08:41:57.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea8154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 0x7f2ea813d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8169f80 con 0x7f2ea806b170 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8169e00 con 0x7f2ea80cd3a0 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea816a100 con 0x7f2ea80cdbf0 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36888/0 (socket says 172.21.15.28:36888) 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea6a00640 1 -- 172.21.15.28:0/1292510260 learned_addr learned my addr 172.21.15.28:0/1292510260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2ea7400640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 0x7f2ea813d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.068+0000 7f2e9e000640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea8154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 -- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 msgr2=0x7f2ea813d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 0x7f2ea813d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 -- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 msgr2=0x7f2ea8154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea8154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 -- 172.21.15.28:0/1292510260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8154cd0 con 0x7f2ea806b170 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea7400640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 0x7f2ea813d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea6a00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800cbf0 tx=0x7f2e9800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98017070 con 0x7f2ea806b170 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9800ad50 con 0x7f2ea806b170 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98012620 con 0x7f2ea806b170 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea815ef00 con 0x7f2ea806b170 2024-03-21T08:41:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea813c6d0 con 0x7f2ea806b170 2024-03-21T08:41:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea8075270 con 0x7f2ea806b170 2024-03-21T08:41:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2e9801a070 con 0x7f2ea806b170 2024-03-21T08:41:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 0x7f2e8406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2e98086110 con 0x7f2ea806b170 2024-03-21T08:41:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.072+0000 7f2e9e000640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 0x7f2e8406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.076+0000 7f2e9e000640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 0x7f2e8406a010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2e90001720 tx=0x7f2e90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.076+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2e98056020 con 0x7f2ea806b170 2024-03-21T08:41:57.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ea8072de0 con 0x7f2ea806b170 2024-03-21T08:41:57.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2e9f400640 1 -- 172.21.15.28:0/1292510260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f2e9805a0c0 con 0x7f2ea806b170 2024-03-21T08:41:57.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:57.246 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 msgr2=0x7f2e8406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 0x7f2e8406a010 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2e90001720 tx=0x7f2e90009040 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 msgr2=0x7f2ea81553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800cbf0 tx=0x7f2e9800a430 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 shutdown_connections 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e84067b50 0x7f2e8406a010 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2ea80cdbf0 0x7f2ea813d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2ea80cd3a0 0x7f2ea8154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1292510260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea806b170 0x7f2ea81553c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 >> 172.21.15.28:0/1292510260 conn(0x7f2ea806e920 msgr2=0x7f2ea80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 shutdown_connections 2024-03-21T08:41:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.244+0000 7f2ea7e00640 1 -- 172.21.15.28:0/1292510260 wait complete. 2024-03-21T08:41:57.268 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:57.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 Processor -- start 2024-03-21T08:41:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 -- start start 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e7806d3a0 0x7f9e780d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9e780d0e00 0x7f9e780ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e780b48a0 con 0x7f9e7815df10 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e780cc0a0 con 0x7f9e780d0e00 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e7d800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780cc940 con 0x7f9e7806d3a0 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9e780d0e00 0x7f9e780ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9e780d0e00 0x7f9e780ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36904/0 (socket says 172.21.15.28:36904) 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 -- 172.21.15.28:0/44058026 learned_addr learned my addr 172.21.15.28:0/44058026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e6fe00640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e7806d3a0 0x7f9e780d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e76a00640 1 --1- 172.21.15.28:0/44058026 >> v1:172.21.15.151:6789/0 conn(0x7f9e7815df10 0x7f9e7806cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-03-21T08:41:57.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 -- 172.21.15.28:0/44058026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e7806d3a0 msgr2=0x7f9e780d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:57.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e7806d3a0 0x7f9e780d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 -- 172.21.15.28:0/44058026 >> v1:172.21.15.151:6789/0 conn(0x7f9e7815df10 legacy=0x7f9e7806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.412+0000 7f9e77400640 1 -- 172.21.15.28:0/44058026 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780d1620 con 0x7f9e780d0e00 2024-03-21T08:41:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e77400640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780d0e00 0x7f9e780ccdd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009e50 tx=0x7f9e7001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43ff1e33346d0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e76000640 1 -- 172.21.15.28:0/44058026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e700a5750 con 0x7f9e780d0e00 2024-03-21T08:41:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e76000640 1 -- 172.21.15.28:0/44058026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e700a58f0 con 0x7f9e780d0e00 2024-03-21T08:41:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e76000640 1 -- 172.21.15.28:0/44058026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e700a5cc0 con 0x7f9e780d0e00 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- 172.21.15.28:0/44058026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780d0e00 msgr2=0x7f9e780ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780d0e00 0x7f9e780ccdd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009e50 tx=0x7f9e7001c9a0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- 172.21.15.28:0/44058026 shutdown_connections 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e780d0e00 0x7f9e780ccdd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 --2- 172.21.15.28:0/44058026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e7806d3a0 0x7f9e780d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- 172.21.15.28:0/44058026 >> 172.21.15.28:0/44058026 conn(0x7f9e780b3540 msgr2=0x7f9e780b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- 172.21.15.28:0/44058026 shutdown_connections 2024-03-21T08:41:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- 172.21.15.28:0/44058026 wait complete. 2024-03-21T08:41:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 Processor -- start 2024-03-21T08:41:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.416+0000 7f9e7d800640 1 -- start start 2024-03-21T08:41:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e780d0e00 0x7f9e780c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e7815df10 0x7f9e78070860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780cdca0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780cdb20 con 0x7f9e7815df10 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780cde20 con 0x7f9e780d0e00 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36914/0 (socket says 172.21.15.28:36914) 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 -- 172.21.15.28:0/2379708164 learned_addr learned my addr 172.21.15.28:0/2379708164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e6fe00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e780d0e00 0x7f9e780c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e77400640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e7815df10 0x7f9e78070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 -- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e780d0e00 msgr2=0x7f9e780c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e780d0e00 0x7f9e780c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 -- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e7815df10 msgr2=0x7f9e78070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e7815df10 0x7f9e78070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 -- 172.21.15.28:0/2379708164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7815a130 con 0x7f9e7806d3a0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e76a00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c450 tx=0x7f9e6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e680074e0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e68007680 con 0x7f9e7806d3a0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68004a30 con 0x7f9e7806d3a0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7815a410 con 0x7f9e7806d3a0 2024-03-21T08:41:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7815a9d0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7804f4f0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.420+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9e6801a020 con 0x7f9e7806d3a0 2024-03-21T08:41:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.424+0000 7f9e6ea00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 0x7f9e5406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.424+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f9e680859d0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.424+0000 7f9e6fe00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 0x7f9e5406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.424+0000 7f9e6fe00640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 0x7f9e5406a010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e7815d930 tx=0x7f9e60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.428+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9e780b48a0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e780672c0 con 0x7f9e7806d3a0 2024-03-21T08:41:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e6ea00640 1 -- 172.21.15.28:0/2379708164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e68055970 con 0x7f9e7806d3a0 2024-03-21T08:41:57.593 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 msgr2=0x7f9e5406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 0x7f9e5406a010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e7815d930 tx=0x7f9e60009190 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 msgr2=0x7f9e78159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c450 tx=0x7f9e6800c910 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 shutdown_connections 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e7815df10 0x7f9e78070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e54067b50 0x7f9e5406a010 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e780d0e00 0x7f9e780c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 --2- 172.21.15.28:0/2379708164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7806d3a0 0x7f9e78159630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 >> 172.21.15.28:0/2379708164 conn(0x7f9e780b3540 msgr2=0x7f9e780b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 shutdown_connections 2024-03-21T08:41:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.592+0000 7f9e7d800640 1 -- 172.21.15.28:0/2379708164 wait complete. 2024-03-21T08:41:57.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 Processor -- start 2024-03-21T08:41:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 -- start start 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faae4144690 0x7faae40b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae40723e0 con 0x7faae41442b0 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae40b2ad0 con 0x7faae4144690 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faaeba00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae40b2d20 con 0x7faae4071a80 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae9a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae9a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:55564/0 (socket says 172.21.15.28:55564) 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae9a00640 1 -- 172.21.15.28:0/1742024521 learned_addr learned my addr 172.21.15.28:0/1742024521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae3e00640 1 --2- 172.21.15.28:0/1742024521 >> v2:172.21.15.151:3300/0 conn(0x7faae4144690 0x7faae40b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae3e00640 1 -- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 msgr2=0x7faae4071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae3e00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae3e00640 1 -- 172.21.15.28:0/1742024521 >> v1:172.21.15.151:6789/0 conn(0x7faae41442b0 legacy=0x7faae40aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae3e00640 1 -- 172.21.15.28:0/1742024521 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae40b3000 con 0x7faae4144690 2024-03-21T08:41:57.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.756+0000 7faae9a00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:57.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faae3e00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae40b3dc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faad00097c0 tx=0x7faad001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aadbead5ecdacbc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faae3400640 1 -- 172.21.15.28:0/1742024521 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00a7070 con 0x7faae4144690 2024-03-21T08:41:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faae3400640 1 -- 172.21.15.28:0/1742024521 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad00b0070 con 0x7faae4144690 2024-03-21T08:41:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faae3400640 1 -- 172.21.15.28:0/1742024521 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00ab890 con 0x7faae4144690 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 -- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 msgr2=0x7faae40b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae40b3dc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faad00097c0 tx=0x7faad001c470 comp rx=0 tx=0).stop 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 -- 172.21.15.28:0/1742024521 shutdown_connections 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae4071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 --2- 172.21.15.28:0/1742024521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae40b3dc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 -- 172.21.15.28:0/1742024521 >> 172.21.15.28:0/1742024521 conn(0x7faae40674f0 msgr2=0x7faae406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 -- 172.21.15.28:0/1742024521 shutdown_connections 2024-03-21T08:41:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.760+0000 7faaeba00640 1 -- 172.21.15.28:0/1742024521 wait complete. 2024-03-21T08:41:57.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 Processor -- start 2024-03-21T08:41:57.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 -- start start 2024-03-21T08:41:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae413b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae4162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40af470 con 0x7faae41442b0 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40af2f0 con 0x7faae4144690 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae406e2b0 con 0x7faae4071a80 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae9a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae4162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 -- 172.21.15.28:0/1901672321 learned_addr learned my addr 172.21.15.28:0/1901672321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae9000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 -- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 msgr2=0x7faae413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 -- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 msgr2=0x7faae4162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae4162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 -- 172.21.15.28:0/1901672321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae413c810 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae3e00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faad001a0a0 tx=0x7faad001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00bf610 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad00bf7d0 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00bfbc0 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.764+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae413caf0 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae413d0b0 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae4073430 con 0x7faae41442b0 2024-03-21T08:41:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faad00bd020 con 0x7faae41442b0 2024-03-21T08:41:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faae0c00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 0x7faac4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7faad0120f20 con 0x7faae41442b0 2024-03-21T08:41:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faae9000640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 0x7faac4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.768+0000 7faae9000640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 0x7faac4069fe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faad8001810 tx=0x7faad8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.772+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faad0016090 con 0x7faae41442b0 2024-03-21T08:41:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.932+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faae404f4f0 con 0x7faae41442b0 2024-03-21T08:41:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.932+0000 7faae0c00640 1 -- 172.21.15.28:0/1901672321 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7faad00f0f40 con 0x7faae41442b0 2024-03-21T08:41:57.935 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 msgr2=0x7faac4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 0x7faac4069fe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faad8001810 tx=0x7faad8009040 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 msgr2=0x7faae413bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faad001a0a0 tx=0x7faad001acb0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 shutdown_connections 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faae4144690 0x7faae4162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae41442b0 0x7faae413bea0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faac4067b20 0x7faac4069fe0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 --2- 172.21.15.28:0/1901672321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faae4071a80 0x7faae413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 >> 172.21.15.28:0/1901672321 conn(0x7faae40674f0 msgr2=0x7faae406e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 shutdown_connections 2024-03-21T08:41:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:57.936+0000 7faaeba00640 1 -- 172.21.15.28:0/1901672321 wait complete. 2024-03-21T08:41:57.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:41:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 Processor -- start 2024-03-21T08:41:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 -- start start 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd474162c90 0x7fd4740b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47406ea00 con 0x7fd474163070 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47406dc40 con 0x7fd474163500 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd47b600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4740b5000 con 0x7fd474162c90 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd474163070 0x7fd474071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41870/0 (socket says 172.21.15.28:41870) 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473e00640 1 -- 172.21.15.28:0/2951104523 learned_addr learned my addr 172.21.15.28:0/2951104523 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd478c00640 1 --2- 172.21.15.28:0/2951104523 >> v2:172.21.15.151:3300/0 conn(0x7fd474162c90 0x7fd4740b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762973700 0 0) 0x7fd47406ea00 con 0x7fd474163070 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd45c003530 con 0x7fd474163070 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485009344 0 0) 0x7fd47406dc40 con 0x7fd474163500 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47406ea00 con 0x7fd474163500 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd478c00640 1 -- 172.21.15.28:0/2951104523 >> v1:172.21.15.151:6789/0 conn(0x7fd474163500 legacy=0x7fd4740b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd478c00640 1 -- 172.21.15.28:0/2951104523 >> v1:172.21.15.28:6789/0 conn(0x7fd474163070 legacy=0x7fd474071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.100+0000 7fd478c00640 1 -- 172.21.15.28:0/2951104523 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47406df20 con 0x7fd474162c90 2024-03-21T08:41:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd478c00640 1 --2- 172.21.15.28:0/2951104523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd4740b4890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd4680097c0 tx=0x7fd46801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228989eb5d55c5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4680b0070 con 0x7fd474162c90 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4680ab600 con 0x7fd474162c90 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd473400640 1 -- 172.21.15.28:0/2951104523 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4680ab9b0 con 0x7fd474162c90 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 -- 172.21.15.28:0/2951104523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 msgr2=0x7fd4740b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 --2- 172.21.15.28:0/2951104523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd4740b4890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd4680097c0 tx=0x7fd46801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 -- 172.21.15.28:0/2951104523 shutdown_connections 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 --2- 172.21.15.28:0/2951104523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd4740b4890 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 -- 172.21.15.28:0/2951104523 >> 172.21.15.28:0/2951104523 conn(0x7fd4740674f0 msgr2=0x7fd47406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 -- 172.21.15.28:0/2951104523 shutdown_connections 2024-03-21T08:41:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.104+0000 7fd47b600640 1 -- 172.21.15.28:0/2951104523 wait complete. 2024-03-21T08:41:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 Processor -- start 2024-03-21T08:41:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- start start 2024-03-21T08:41:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd47414cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd474163070 0x7fd47415d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474072310 con 0x7fd47415d920 2024-03-21T08:41:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474072190 con 0x7fd474162c90 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741700f0 con 0x7fd474163070 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd478c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd47414cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36922/0 (socket says 172.21.15.28:36922) 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 -- 172.21.15.28:0/3432870750 learned_addr learned my addr 172.21.15.28:0/3432870750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd473e00640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd474163070 0x7fd47415d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 -- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd474163070 msgr2=0x7fd47415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd474163070 0x7fd47415d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 -- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 msgr2=0x7fd47414cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd47414cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 -- 172.21.15.28:0/3432870750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd474160e60 con 0x7fd47415d920 2024-03-21T08:41:58.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd478c00640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd47414cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd479600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00c760 tx=0x7fd46c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c003b80 con 0x7fd47415d920 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c003d20 con 0x7fd47415d920 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00aa90 con 0x7fd47415d920 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474161140 con 0x7fd47415d920 2024-03-21T08:41:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.108+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47415cec0 con 0x7fd47415d920 2024-03-21T08:41:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474073430 con 0x7fd47415d920 2024-03-21T08:41:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd46c00ac30 con 0x7fd47415d920 2024-03-21T08:41:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd470c00640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fd46c086080 con 0x7fd47415d920 2024-03-21T08:41:58.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd478c00640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.112+0000 7fd478c00640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4680ad000 tx=0x7fd468008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.116+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd46c055f90 con 0x7fd47415d920 2024-03-21T08:41:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.280+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd47404f4f0 con 0x7fd47415d920 2024-03-21T08:41:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.280+0000 7fd470c00640 1 -- 172.21.15.28:0/3432870750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd46c05a030 con 0x7fd47415d920 2024-03-21T08:41:58.283 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:41:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 msgr2=0x7fd45406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4680ad000 tx=0x7fd468008040 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 msgr2=0x7fd474155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00c760 tx=0x7fd46c00cc20 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 shutdown_connections 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47415d920 0x7fd474155350 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd474163070 0x7fd47415d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 --2- 172.21.15.28:0/3432870750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd474162c90 0x7fd47414cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 >> 172.21.15.28:0/3432870750 conn(0x7fd4740674f0 msgr2=0x7fd474157030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 shutdown_connections 2024-03-21T08:41:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.284+0000 7fd47b600640 1 -- 172.21.15.28:0/3432870750 wait complete. 2024-03-21T08:41:58.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:41:58.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.444+0000 7efc7f200640 1 Processor -- start 2024-03-21T08:41:58.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.444+0000 7efc7f200640 1 -- start start 2024-03-21T08:41:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc780aedd0 0x7efc7815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc78072310 0x7efc780727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc780cee20 con 0x7efc780b2de0 2024-03-21T08:41:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780cbdf0 con 0x7efc780aedd0 2024-03-21T08:41:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cf100 con 0x7efc78072310 2024-03-21T08:41:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc780aedd0 0x7efc7815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc780aedd0 0x7efc7815e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36928/0 (socket says 172.21.15.28:36928) 2024-03-21T08:41:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 -- 172.21.15.28:0/748325052 learned_addr learned my addr 172.21.15.28:0/748325052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc7d200640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc78072310 0x7efc780727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 -- 172.21.15.28:0/748325052 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc78072310 msgr2=0x7efc780727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc78072310 0x7efc780727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 -- 172.21.15.28:0/748325052 >> v1:172.21.15.28:6789/0 conn(0x7efc780b2de0 legacy=0x7efc780ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.448+0000 7efc77e00640 1 -- 172.21.15.28:0/748325052 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780cf3e0 con 0x7efc780aedd0 2024-03-21T08:41:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc77e00640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc7815e850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efc68009e50 tx=0x7efc6801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1724c1ddffd4fd90 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc77400640 1 -- 172.21.15.28:0/748325052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc680a5680 con 0x7efc780aedd0 2024-03-21T08:41:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc77400640 1 -- 172.21.15.28:0/748325052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc680a5820 con 0x7efc780aedd0 2024-03-21T08:41:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc77400640 1 -- 172.21.15.28:0/748325052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc680a5bf0 con 0x7efc780aedd0 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- 172.21.15.28:0/748325052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 msgr2=0x7efc7815e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc7815e850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efc68009e50 tx=0x7efc6801ca30 comp rx=0 tx=0).stop 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- 172.21.15.28:0/748325052 shutdown_connections 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc78072310 0x7efc780727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 --2- 172.21.15.28:0/748325052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc7815e850 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- 172.21.15.28:0/748325052 >> 172.21.15.28:0/748325052 conn(0x7efc780674f0 msgr2=0x7efc7806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- 172.21.15.28:0/748325052 shutdown_connections 2024-03-21T08:41:58.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- 172.21.15.28:0/748325052 wait complete. 2024-03-21T08:41:58.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 Processor -- start 2024-03-21T08:41:58.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.452+0000 7efc7f200640 1 -- start start 2024-03-21T08:41:58.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 0x7efc78159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc780b2de0 0x7efc7815b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cc690 con 0x7efc780aedd0 2024-03-21T08:41:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cc510 con 0x7efc78072310 2024-03-21T08:41:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cc810 con 0x7efc780b2de0 2024-03-21T08:41:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36942/0 (socket says 172.21.15.28:36942) 2024-03-21T08:41:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 -- 172.21.15.28:0/1099220322 learned_addr learned my addr 172.21.15.28:0/1099220322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc77e00640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 0x7efc78159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7d200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc780b2de0 0x7efc7815b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 -- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc780b2de0 msgr2=0x7efc7815b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc780b2de0 0x7efc7815b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 -- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 msgr2=0x7efc78159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 0x7efc78159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 -- 172.21.15.28:0/1099220322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78138490 con 0x7efc780aedd0 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc77e00640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 0x7efc78159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc6f400640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c370 tx=0x7efc6000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc600052d0 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60005470 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60007260 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc78138770 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc78138d30 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc78073430 con 0x7efc780aedd0 2024-03-21T08:41:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.456+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7efc60007400 con 0x7efc780aedd0 2024-03-21T08:41:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.460+0000 7efc74c00640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 0x7efc5406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.460+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7efc60085750 con 0x7efc780aedd0 2024-03-21T08:41:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.460+0000 7efc77e00640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 0x7efc5406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.460+0000 7efc77e00640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 0x7efc5406a010 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efc680ae000 tx=0x7efc680a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.464+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efc60055660 con 0x7efc780aedd0 2024-03-21T08:41:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.624+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7efc7804f4f0 con 0x7efc780aedd0 2024-03-21T08:41:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.624+0000 7efc74c00640 1 -- 172.21.15.28:0/1099220322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7efc60059700 con 0x7efc780aedd0 2024-03-21T08:41:58.627 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 msgr2=0x7efc5406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 0x7efc5406a010 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efc680ae000 tx=0x7efc680a7070 comp rx=0 tx=0).stop 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 msgr2=0x7efc78141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c370 tx=0x7efc6000c830 comp rx=0 tx=0).stop 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 shutdown_connections 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7efc54067b50 0x7efc5406a010 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7efc780b2de0 0x7efc7815b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc780aedd0 0x7efc78141f40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 --2- 172.21.15.28:0/1099220322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc78072310 0x7efc78159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 >> 172.21.15.28:0/1099220322 conn(0x7efc780674f0 msgr2=0x7efc7813d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 shutdown_connections 2024-03-21T08:41:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.628+0000 7efc7f200640 1 -- 172.21.15.28:0/1099220322 wait complete. 2024-03-21T08:41:58.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:41:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 Processor -- start 2024-03-21T08:41:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 -- start start 2024-03-21T08:41:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01d00ce6e0 0x7f01d00d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d015e5a0 con 0x7f01d015a7d0 2024-03-21T08:41:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d015e880 con 0x7f01d00d0b90 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d00d0fe0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01d00ce6e0 0x7f01d00d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01d00ce6e0 0x7f01d00d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36948/0 (socket says 172.21.15.28:36948) 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01c7400640 1 -- 172.21.15.28:0/719877299 learned_addr learned my addr 172.21.15.28:0/719877299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.788+0000 7f01d5200640 1 --1- 172.21.15.28:0/719877299 >> v1:172.21.15.151:6789/0 conn(0x7f01d00d0b90 0x7f01d00b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:59036/0 (socket says 172.21.15.28:59036) 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01c7400640 1 -- 172.21.15.28:0/719877299 >> v1:172.21.15.151:6789/0 conn(0x7f01d00d0b90 legacy=0x7f01d00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01c7400640 1 -- 172.21.15.28:0/719877299 >> v1:172.21.15.28:6789/0 conn(0x7f01d015a7d0 legacy=0x7f01d0070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01c7400640 1 -- 172.21.15.28:0/719877299 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d015eb60 con 0x7f01d00ce6e0 2024-03-21T08:41:58.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01cf400640 1 -- 172.21.15.28:0/719877299 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952961927 0 0) 0x7f01d015e880 con 0x7f01d00d0b90 2024-03-21T08:41:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01c7400640 1 --2- 172.21.15.28:0/719877299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d00d0650 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f01b800bb10 tx=0x7f01b801ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c736860a7d0a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01cf400640 1 -- 172.21.15.28:0/719877299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80b4070 con 0x7f01d00ce6e0 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01cf400640 1 -- 172.21.15.28:0/719877299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b80a7970 con 0x7f01d00ce6e0 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01cf400640 1 -- 172.21.15.28:0/719877299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80a7d40 con 0x7f01d00ce6e0 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01d5c00640 1 -- 172.21.15.28:0/719877299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 msgr2=0x7f01d00d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01d5c00640 1 --2- 172.21.15.28:0/719877299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d00d0650 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f01b800bb10 tx=0x7f01b801ea30 comp rx=0 tx=0).stop 2024-03-21T08:41:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01d5c00640 1 -- 172.21.15.28:0/719877299 shutdown_connections 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01d5c00640 1 --2- 172.21.15.28:0/719877299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d00d0650 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.792+0000 7f01d5c00640 1 -- 172.21.15.28:0/719877299 >> 172.21.15.28:0/719877299 conn(0x7f01d00b23d0 msgr2=0x7f01d00b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- 172.21.15.28:0/719877299 shutdown_connections 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- 172.21.15.28:0/719877299 wait complete. 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 Processor -- start 2024-03-21T08:41:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- start start 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01d00d0b90 0x7f01d014d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 0x7f01d0151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00cd040 con 0x7f01d00ce6e0 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00ccec0 con 0x7f01d00d0b90 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d006b170 con 0x7f01d015a7d0 2024-03-21T08:41:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 -- 172.21.15.28:0/1923569937 learned_addr learned my addr 172.21.15.28:0/1923569937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01c7400640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01d00d0b90 0x7f01d014d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5200640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 0x7f01d0151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 -- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 msgr2=0x7f01d0151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 0x7f01d0151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 -- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01d00d0b90 msgr2=0x7f01d014d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01d00d0b90 0x7f01d014d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 -- 172.21.15.28:0/1923569937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0152220 con 0x7f01d00ce6e0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5200640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 0x7f01d0151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01cfe00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c870 tx=0x7f01c000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0017070 con 0x7f01d00ce6e0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c0003db0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c00052b0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d014e3d0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.796+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d014e990 con 0x7f01d00ce6e0 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d004f410 con 0x7f01d00ce6e0 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f01c001a070 con 0x7f01d00ce6e0 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01ccc00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 0x7f01ac069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f01c0085ab0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01c7400640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 0x7f01ac069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:58.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.800+0000 7f01c7400640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 0x7f01ac069fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f01b80b1000 tx=0x7f01b80a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:58.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.804+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f01c00559c0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f01d00704d0 con 0x7f01d00ce6e0 2024-03-21T08:41:58.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01ccc00640 1 -- 172.21.15.28:0/1923569937 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f01c0059a60 con 0x7f01d00ce6e0 2024-03-21T08:41:58.990 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:41:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 msgr2=0x7f01ac069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 0x7f01ac069fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f01b80b1000 tx=0x7f01b80a9070 comp rx=0 tx=0).stop 2024-03-21T08:41:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 msgr2=0x7f01d014f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c870 tx=0x7f01c000cd30 comp rx=0 tx=0).stop 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 shutdown_connections 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f01ac067b20 0x7f01ac069fe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01d015a7d0 0x7f01d0151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01d00d0b90 0x7f01d014d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 --2- 172.21.15.28:0/1923569937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01d00ce6e0 0x7f01d014f470 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.988+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 >> 172.21.15.28:0/1923569937 conn(0x7f01d00b23d0 msgr2=0x7f01d00b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.992+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 shutdown_connections 2024-03-21T08:41:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:58.992+0000 7f01d5c00640 1 -- 172.21.15.28:0/1923569937 wait complete. 2024-03-21T08:41:59.011 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:41:59.011 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:41:59.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:41:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.204+0000 7f8694a00640 1 Processor -- start 2024-03-21T08:41:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.204+0000 7f8694a00640 1 -- start start 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8694a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f86900ce490 0x7f869015e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8694a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f869015ba70 0x7f86900ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8694a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869015bf00 con 0x7f8690071980 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8694a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900cedb0 con 0x7f869015ba70 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8694a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869015ebf0 con 0x7f86900ce490 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f869015ba70 0x7f86900ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f869015ba70 0x7f86900ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36970/0 (socket says 172.21.15.28:36970) 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 -- 172.21.15.28:0/543408620 learned_addr learned my addr 172.21.15.28:0/543408620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:59.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f868de00640 1 --2- 172.21.15.28:0/543408620 >> v2:172.21.15.151:3300/0 conn(0x7f86900ce490 0x7f869015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 -- 172.21.15.28:0/543408620 >> v2:172.21.15.151:3300/0 conn(0x7f86900ce490 msgr2=0x7f869015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:59.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 --2- 172.21.15.28:0/543408620 >> v2:172.21.15.151:3300/0 conn(0x7f86900ce490 0x7f869015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 -- 172.21.15.28:0/543408620 >> v1:172.21.15.151:6789/0 conn(0x7f8690071980 legacy=0x7f8690071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.208+0000 7f8687e00640 1 -- 172.21.15.28:0/543408620 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869015eed0 con 0x7f869015ba70 2024-03-21T08:41:59.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8687e00640 1 --2- 172.21.15.28:0/543408620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869015ba70 0x7f86900ce870 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f86900730c0 tx=0x7f867801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955b26e9d1456263 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f868d400640 1 -- 172.21.15.28:0/543408620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780b2070 con 0x7f869015ba70 2024-03-21T08:41:59.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f868d400640 1 -- 172.21.15.28:0/543408620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86780ad680 con 0x7f869015ba70 2024-03-21T08:41:59.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f868d400640 1 -- 172.21.15.28:0/543408620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780ada50 con 0x7f869015ba70 2024-03-21T08:41:59.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- 172.21.15.28:0/543408620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869015ba70 msgr2=0x7f86900ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 --2- 172.21.15.28:0/543408620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869015ba70 0x7f86900ce870 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f86900730c0 tx=0x7f867801c490 comp rx=0 tx=0).stop 2024-03-21T08:41:59.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- 172.21.15.28:0/543408620 shutdown_connections 2024-03-21T08:41:59.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 --2- 172.21.15.28:0/543408620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869015ba70 0x7f86900ce870 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 --2- 172.21.15.28:0/543408620 >> v2:172.21.15.151:3300/0 conn(0x7f86900ce490 0x7f869015e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- 172.21.15.28:0/543408620 >> 172.21.15.28:0/543408620 conn(0x7f86900cc230 msgr2=0x7f86900cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:59.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- 172.21.15.28:0/543408620 shutdown_connections 2024-03-21T08:41:59.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- 172.21.15.28:0/543408620 wait complete. 2024-03-21T08:41:59.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 Processor -- start 2024-03-21T08:41:59.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.212+0000 7f8694a00640 1 -- start start 2024-03-21T08:41:59.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 0x7f869014ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f869015ba70 0x7f86901521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901681c0 con 0x7f86900ce490 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690155390 con 0x7f869015ba70 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8690168340 con 0x7f8690071980 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f868de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 0x7f869014ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36978/0 (socket says 172.21.15.28:36978) 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 -- 172.21.15.28:0/4156855917 learned_addr learned my addr 172.21.15.28:0/4156855917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f868de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 0x7f869014ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:55590/0 (socket says 172.21.15.28:55590) 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f868e800640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f869015ba70 0x7f86901521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 -- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 msgr2=0x7f869014ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 0x7f869014ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 -- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f869015ba70 msgr2=0x7f86901521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f869015ba70 0x7f86901521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 -- 172.21.15.28:0/4156855917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869014d5d0 con 0x7f86900ce490 2024-03-21T08:41:59.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8687e00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f867801a0c0 tx=0x7f867801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780b2040 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86780ad9b0 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8690157190 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780add80 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86901563d0 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869004f410 con 0x7f86900ce490 2024-03-21T08:41:59.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.216+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f86780bf030 con 0x7f86900ce490 2024-03-21T08:41:59.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.220+0000 7f8686000640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 0x7f866c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.220+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f867811f770 con 0x7f86900ce490 2024-03-21T08:41:59.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.220+0000 7f868de00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 0x7f866c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.220+0000 7f868de00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 0x7f866c06a010 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8690072210 tx=0x7f8680009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.224+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f869014d5d0 con 0x7f86900ce490 2024-03-21T08:41:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.384+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8690072c10 con 0x7f86900ce490 2024-03-21T08:41:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8686000640 1 -- 172.21.15.28:0/4156855917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f8678016110 con 0x7f86900ce490 2024-03-21T08:41:59.389 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:41:59.389 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:41:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 msgr2=0x7f866c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 0x7f866c06a010 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8690072210 tx=0x7f8680009190 comp rx=0 tx=0).stop 2024-03-21T08:41:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 msgr2=0x7f8690151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f867801a0c0 tx=0x7f867801acd0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 shutdown_connections 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f869015ba70 0x7f86901521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86900ce490 0x7f8690151c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f866c067b50 0x7f866c06a010 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 --2- 172.21.15.28:0/4156855917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8690071980 0x7f869014ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 >> 172.21.15.28:0/4156855917 conn(0x7f86900cc230 msgr2=0x7f86900cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 shutdown_connections 2024-03-21T08:41:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.388+0000 7f8694a00640 1 -- 172.21.15.28:0/4156855917 wait complete. 2024-03-21T08:41:59.410 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:41:59.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:41:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 Processor -- start 2024-03-21T08:41:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 -- start start 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f374015aa30 0x7f374015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37400ce590 0x7f37400d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f374015b350 0x7f37400d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37400cf930 con 0x7f374015b350 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374015be50 con 0x7f374015aa30 2024-03-21T08:41:59.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3746400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374015b890 con 0x7f37400ce590 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f374015b350 0x7f37400d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373f400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f374015aa30 0x7f374015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f3737e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37400ce590 0x7f37400d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f374015b350 0x7f37400d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36986/0 (socket says 172.21.15.28:36986) 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 -- 172.21.15.28:0/1274976219 learned_addr learned my addr 172.21.15.28:0/1274976219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 -- 172.21.15.28:0/1274976219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37400ce590 msgr2=0x7f37400d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 --2- 172.21.15.28:0/1274976219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37400ce590 0x7f37400d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 -- 172.21.15.28:0/1274976219 >> v2:172.21.15.151:3300/0 conn(0x7f374015aa30 msgr2=0x7f374015ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 --2- 172.21.15.28:0/1274976219 >> v2:172.21.15.151:3300/0 conn(0x7f374015aa30 0x7f374015ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.552+0000 7f373fe00640 1 -- 172.21.15.28:0/1274976219 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374015bb70 con 0x7f374015b350 2024-03-21T08:41:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f373fe00640 1 --2- 172.21.15.28:0/1274976219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015b350 0x7f37400d0ad0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f37400736b0 tx=0x7f373801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd87d80c8cfa755 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f373ea00640 1 -- 172.21.15.28:0/1274976219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380a56d0 con 0x7f374015b350 2024-03-21T08:41:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f373ea00640 1 -- 172.21.15.28:0/1274976219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37380a5870 con 0x7f374015b350 2024-03-21T08:41:59.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f373ea00640 1 -- 172.21.15.28:0/1274976219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380a5c40 con 0x7f374015b350 2024-03-21T08:41:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 -- 172.21.15.28:0/1274976219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015b350 msgr2=0x7f37400d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 --2- 172.21.15.28:0/1274976219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015b350 0x7f37400d0ad0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f37400736b0 tx=0x7f373801c960 comp rx=0 tx=0).stop 2024-03-21T08:41:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 -- 172.21.15.28:0/1274976219 shutdown_connections 2024-03-21T08:41:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 --2- 172.21.15.28:0/1274976219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015b350 0x7f37400d0ad0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 --2- 172.21.15.28:0/1274976219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f37400ce590 0x7f37400d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 --2- 172.21.15.28:0/1274976219 >> v2:172.21.15.151:3300/0 conn(0x7f374015aa30 0x7f374015ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 -- 172.21.15.28:0/1274976219 >> 172.21.15.28:0/1274976219 conn(0x7f3740099a30 msgr2=0x7f3740099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 -- 172.21.15.28:0/1274976219 shutdown_connections 2024-03-21T08:41:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.556+0000 7f3746400640 1 -- 172.21.15.28:0/1274976219 wait complete. 2024-03-21T08:41:59.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 Processor -- start 2024-03-21T08:41:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- start start 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 0x7f374014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f374015b350 0x7f37400cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374015e530 con 0x7f374015aa30 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400ce140 con 0x7f37400ce590 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374015e6b0 con 0x7f374015b350 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37000/0 (socket says 172.21.15.28:37000) 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 -- 172.21.15.28:0/1082549802 learned_addr learned my addr 172.21.15.28:0/1082549802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f373f400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 0x7f374014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f373fe00640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f374015b350 0x7f37400cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 -- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f374015b350 msgr2=0x7f37400cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f374015b350 0x7f37400cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 -- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 msgr2=0x7f374014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 0x7f374014abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 -- 172.21.15.28:0/1082549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37400c95f0 con 0x7f374015aa30 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f373f400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 0x7f374014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737e00640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f372800e900 tx=0x7f372800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728017070 con 0x7f374015aa30 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372800ca00 con 0x7f374015aa30 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800cdd0 con 0x7f374015aa30 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37400c98d0 con 0x7f374015aa30 2024-03-21T08:41:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.560+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37400ca170 con 0x7f374015aa30 2024-03-21T08:41:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374009ea60 con 0x7f374015aa30 2024-03-21T08:41:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f372801a070 con 0x7f374015aa30 2024-03-21T08:41:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f3737400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 0x7f371c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f3728086690 con 0x7f374015aa30 2024-03-21T08:41:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f373f400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 0x7f371c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.564+0000 7f373f400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 0x7f371c06a010 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3730009b30 tx=0x7f3730008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.568+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f37280565a0 con 0x7f374015aa30 2024-03-21T08:41:59.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.732+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f374004f4f0 con 0x7f374015aa30 2024-03-21T08:41:59.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.732+0000 7f3737400640 1 -- 172.21.15.28:0/1082549802 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f372805a640 con 0x7f374015aa30 2024-03-21T08:41:59.734 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:41:59.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 msgr2=0x7f371c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 0x7f371c06a010 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3730009b30 tx=0x7f3730008040 comp rx=0 tx=0).stop 2024-03-21T08:41:59.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 msgr2=0x7f37400c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f372800e900 tx=0x7f372800edc0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 shutdown_connections 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f371c067b50 0x7f371c06a010 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f374015b350 0x7f37400cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f374015aa30 0x7f37400c8ca0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 --2- 172.21.15.28:0/1082549802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f37400ce590 0x7f374014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 >> 172.21.15.28:0/1082549802 conn(0x7f3740099a30 msgr2=0x7f37401599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 shutdown_connections 2024-03-21T08:41:59.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.736+0000 7f3746400640 1 -- 172.21.15.28:0/1082549802 wait complete. 2024-03-21T08:41:59.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:41:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 Processor -- start 2024-03-21T08:41:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 -- start start 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 0x7f9a98168990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a980b48a0 con 0x7f9a9806d3a0 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a980d1620 con 0x7f9a9815df10 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a9e600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a980cc0a0 con 0x7f9a980d0fd0 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a8ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9a9806d3a0 0x7f9a980d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41894/0 (socket says 172.21.15.28:41894) 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a8ea00640 1 -- 172.21.15.28:0/1334842973 learned_addr learned my addr 172.21.15.28:0/1334842973 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a97e00640 1 --2- 172.21.15.28:0/1334842973 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 0x7f9a98168990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036159822 0 0) 0x7f9a80001120 con 0x7f9a9806d3a0 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c003530 con 0x7f9a9806d3a0 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144207403 0 0) 0x7f9a980d1620 con 0x7f9a9815df10 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a80001120 con 0x7f9a9815df10 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a97e00640 1 -- 172.21.15.28:0/1334842973 >> v1:172.21.15.151:6789/0 conn(0x7f9a9815df10 legacy=0x7f9a9806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a97e00640 1 -- 172.21.15.28:0/1334842973 >> v1:172.21.15.28:6789/0 conn(0x7f9a9806d3a0 legacy=0x7f9a980d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.900+0000 7f9a97e00640 1 -- 172.21.15.28:0/1334842973 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a980cc2f0 con 0x7f9a980d0fd0 2024-03-21T08:41:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a97e00640 1 --2- 172.21.15.28:0/1334842973 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 0x7f9a98168990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009dc0 tx=0x7f9a9001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec945430c9f8ec53 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900b2070 con 0x7f9a980d0fd0 2024-03-21T08:41:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a900a57c0 con 0x7f9a980d0fd0 2024-03-21T08:41:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a96a00640 1 -- 172.21.15.28:0/1334842973 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900a5b70 con 0x7f9a980d0fd0 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 -- 172.21.15.28:0/1334842973 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 msgr2=0x7f9a98168990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 --2- 172.21.15.28:0/1334842973 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 0x7f9a98168990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009dc0 tx=0x7f9a9001c900 comp rx=0 tx=0).stop 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 -- 172.21.15.28:0/1334842973 shutdown_connections 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 --2- 172.21.15.28:0/1334842973 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a980d0fd0 0x7f9a98168990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 -- 172.21.15.28:0/1334842973 >> 172.21.15.28:0/1334842973 conn(0x7f9a980b3540 msgr2=0x7f9a980b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 -- 172.21.15.28:0/1334842973 shutdown_connections 2024-03-21T08:41:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.904+0000 7f9a9e600640 1 -- 172.21.15.28:0/1334842973 wait complete. 2024-03-21T08:41:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 Processor -- start 2024-03-21T08:41:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- start start 2024-03-21T08:41:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9806d3a0 0x7f9a98158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a980d0fd0 0x7f9a980c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9816b900 con 0x7f9a9815df10 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a980cc940 con 0x7f9a980d0fd0 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9816ba80 con 0x7f9a9806d3a0 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37008/0 (socket says 172.21.15.28:37008) 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 -- 172.21.15.28:0/819871669 learned_addr learned my addr 172.21.15.28:0/819871669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97400640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9806d3a0 0x7f9a98158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a8ea00640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a980d0fd0 0x7f9a980c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 -- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9806d3a0 msgr2=0x7f9a98158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9806d3a0 0x7f9a98158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 -- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a980d0fd0 msgr2=0x7f9a980c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a980d0fd0 0x7f9a980c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 -- 172.21.15.28:0/819871669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a980c8030 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a97e00640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9a9001a530 tx=0x7f9a900196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900bd020 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a900b8470 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a980c8310 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900b2040 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9815fba0 con 0x7f9a9815df10 2024-03-21T08:41:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.908+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9804f4f0 con 0x7f9a9815df10 2024-03-21T08:41:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.912+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9a900c0020 con 0x7f9a9815df10 2024-03-21T08:41:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.912+0000 7f9a8fe00640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 0x7f9a7406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:41:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.912+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f9a90120b50 con 0x7f9a9815df10 2024-03-21T08:41:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.912+0000 7f9a97400640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 0x7f9a7406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:41:59.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.912+0000 7f9a97400640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 0x7f9a7406a010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a880011f0 tx=0x7f9a88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:41:59.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:41:59.916+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9a900c8020 con 0x7f9a9815df10 2024-03-21T08:42:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.084+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a9809a190 con 0x7f9a9815df10 2024-03-21T08:42:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.084+0000 7f9a8fe00640 1 -- 172.21.15.28:0/819871669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a90016580 con 0x7f9a9815df10 2024-03-21T08:42:00.087 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 msgr2=0x7f9a7406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 0x7f9a7406a010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a880011f0 tx=0x7f9a88009040 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 msgr2=0x7f9a980c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9a9001a530 tx=0x7f9a900196a0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 shutdown_connections 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9815df10 0x7f9a980c5de0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a980d0fd0 0x7f9a980c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a74067b50 0x7f9a7406a010 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 --2- 172.21.15.28:0/819871669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9806d3a0 0x7f9a98158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 >> 172.21.15.28:0/819871669 conn(0x7f9a980b3540 msgr2=0x7f9a9809a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 shutdown_connections 2024-03-21T08:42:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.088+0000 7f9a9e600640 1 -- 172.21.15.28:0/819871669 wait complete. 2024-03-21T08:42:00.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:00.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 Processor -- start 2024-03-21T08:42:00.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 -- start start 2024-03-21T08:42:00.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8650067340 0x7f865015bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 0x7f865015c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86500d03a0 con 0x7f86500b12a0 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500b4260 con 0x7f8650067340 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8655400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500af810 con 0x7f86500677a0 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8650067340 0x7f865015bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8650067340 0x7f865015bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37020/0 (socket says 172.21.15.28:37020) 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 -- 172.21.15.28:0/422525686 learned_addr learned my addr 172.21.15.28:0/422525686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f864f200640 1 --2- 172.21.15.28:0/422525686 >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 0x7f865015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 -- 172.21.15.28:0/422525686 >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 msgr2=0x7f865015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 --2- 172.21.15.28:0/422525686 >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 0x7f865015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 -- 172.21.15.28:0/422525686 >> v1:172.21.15.151:6789/0 conn(0x7f86500b12a0 legacy=0x7f86500b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f8647e00640 1 -- 172.21.15.28:0/422525686 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500d0610 con 0x7f8650067340 2024-03-21T08:42:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.256+0000 7f864f200640 1 --2- 172.21.15.28:0/422525686 >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 0x7f865015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8647e00640 1 --2- 172.21.15.28:0/422525686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8650067340 0x7f865015bfc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8650071e00 tx=0x7f863801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6d5769575806d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f864de00640 1 -- 172.21.15.28:0/422525686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86380aa670 con 0x7f8650067340 2024-03-21T08:42:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f864de00640 1 -- 172.21.15.28:0/422525686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86380af070 con 0x7f8650067340 2024-03-21T08:42:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f864de00640 1 -- 172.21.15.28:0/422525686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86380aaa30 con 0x7f8650067340 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- 172.21.15.28:0/422525686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8650067340 msgr2=0x7f865015bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 --2- 172.21.15.28:0/422525686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8650067340 0x7f865015bfc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8650071e00 tx=0x7f863801c4f0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- 172.21.15.28:0/422525686 shutdown_connections 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 --2- 172.21.15.28:0/422525686 >> v2:172.21.15.151:3300/0 conn(0x7f86500677a0 0x7f865015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 --2- 172.21.15.28:0/422525686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8650067340 0x7f865015bfc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- 172.21.15.28:0/422525686 >> 172.21.15.28:0/422525686 conn(0x7f8650099a30 msgr2=0x7f86500b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- 172.21.15.28:0/422525686 shutdown_connections 2024-03-21T08:42:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- 172.21.15.28:0/422525686 wait complete. 2024-03-21T08:42:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 Processor -- start 2024-03-21T08:42:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.260+0000 7f8655400640 1 -- start start 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8650067340 0x7f8650157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500677a0 0x7f8650165860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86500d0f90 con 0x7f86500b12a0 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86500d0e10 con 0x7f86500677a0 2024-03-21T08:42:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8650168950 con 0x7f8650067340 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37034/0 (socket says 172.21.15.28:37034) 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 -- 172.21.15.28:0/1180373456 learned_addr learned my addr 172.21.15.28:0/1180373456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864e800640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8650067340 0x7f8650157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8647e00640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500677a0 0x7f8650165860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 -- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8650067340 msgr2=0x7f8650157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8650067340 0x7f8650157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 -- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500677a0 msgr2=0x7f8650165860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500677a0 0x7f8650165860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 -- 172.21.15.28:0/1180373456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650163ef0 con 0x7f86500b12a0 2024-03-21T08:42:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f864f200640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8648007f20 tx=0x7f864800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864801a070 con 0x7f86500b12a0 2024-03-21T08:42:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8648003ba0 con 0x7f86500b12a0 2024-03-21T08:42:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648004030 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86501641d0 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8650164790 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865009e630 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.264+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f864801d070 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.268+0000 7f8646a00640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 0x7f862c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.268+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f8648086610 con 0x7f86500b12a0 2024-03-21T08:42:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.268+0000 7f864e800640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 0x7f862c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.268+0000 7f864e800640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 0x7f862c069fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8640001770 tx=0x7f8640009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.272+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8648010070 con 0x7f86500b12a0 2024-03-21T08:42:00.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.432+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f865004f4f0 con 0x7f86500b12a0 2024-03-21T08:42:00.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8646a00640 1 -- 172.21.15.28:0/1180373456 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8648056520 con 0x7f86500b12a0 2024-03-21T08:42:00.436 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 msgr2=0x7f862c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 0x7f862c069fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8640001770 tx=0x7f8640009040 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 msgr2=0x7f8650151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8648007f20 tx=0x7f864800d560 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 shutdown_connections 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86500b12a0 0x7f8650151cf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f86500677a0 0x7f8650165860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f862c067b20 0x7f862c069fe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 --2- 172.21.15.28:0/1180373456 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8650067340 0x7f8650157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 >> 172.21.15.28:0/1180373456 conn(0x7f8650099a30 msgr2=0x7f865015ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 shutdown_connections 2024-03-21T08:42:00.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.436+0000 7f8655400640 1 -- 172.21.15.28:0/1180373456 wait complete. 2024-03-21T08:42:00.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 Processor -- start 2024-03-21T08:42:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 -- start start 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f272815bdc0 0x7f27280cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27280cc640 0x7f27280d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27280ccbd0 con 0x7f27280d08f0 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27280cd7e0 con 0x7f27280cc640 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272fa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280cceb0 con 0x7f272815bdc0 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27280cc640 0x7f27280d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272d000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f272815bdc0 0x7f27280cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f27280d08f0 0x7f2728099c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55544/0 (socket says 172.21.15.28:55544) 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f272da00640 1 -- 172.21.15.28:0/1222794046 learned_addr learned my addr 172.21.15.28:0/1222794046 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27280cc640 0x7f27280d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53716/0 (socket says 172.21.15.28:53716) 2024-03-21T08:42:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 -- 172.21.15.28:0/1222794046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f272815bdc0 msgr2=0x7f27280cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 --2- 172.21.15.28:0/1222794046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f272815bdc0 0x7f27280cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 -- 172.21.15.28:0/1222794046 >> v1:172.21.15.28:6789/0 conn(0x7f27280d08f0 legacy=0x7f2728099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.600+0000 7f271fe00640 1 -- 172.21.15.28:0/1222794046 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27280cd190 con 0x7f27280cc640 2024-03-21T08:42:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f271fe00640 1 --2- 172.21.15.28:0/1222794046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280cc640 0x7f27280d03b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43af5bdcccb3af43 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f2727e00640 1 -- 172.21.15.28:0/1222794046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100b1070 con 0x7f27280cc640 2024-03-21T08:42:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f2727e00640 1 -- 172.21.15.28:0/1222794046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100ac600 con 0x7f27280cc640 2024-03-21T08:42:00.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f2727e00640 1 -- 172.21.15.28:0/1222794046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100ac9d0 con 0x7f27280cc640 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 -- 172.21.15.28:0/1222794046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280cc640 msgr2=0x7f27280d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 --2- 172.21.15.28:0/1222794046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280cc640 0x7f27280d03b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001c3e0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 -- 172.21.15.28:0/1222794046 shutdown_connections 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 --2- 172.21.15.28:0/1222794046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280cc640 0x7f27280d03b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 --2- 172.21.15.28:0/1222794046 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f272815bdc0 0x7f27280cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 -- 172.21.15.28:0/1222794046 >> 172.21.15.28:0/1222794046 conn(0x7f27280b3620 msgr2=0x7f27280b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 -- 172.21.15.28:0/1222794046 shutdown_connections 2024-03-21T08:42:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.604+0000 7f272fa00640 1 -- 172.21.15.28:0/1222794046 wait complete. 2024-03-21T08:42:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 Processor -- start 2024-03-21T08:42:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- start start 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 0x7f2728155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f272815bdc0 0x7f272806c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280b1b40 con 0x7f27280d08f0 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272809a5b0 con 0x7f272815bdc0 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280b1cc0 con 0x7f27280cc640 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53726/0 (socket says 172.21.15.28:53726) 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 -- 172.21.15.28:0/3376631035 learned_addr learned my addr 172.21.15.28:0/3376631035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272d000640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 0x7f2728155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272da00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f272815bdc0 0x7f272806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 -- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 msgr2=0x7f2728155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 0x7f2728155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 -- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f272815bdc0 msgr2=0x7f272806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f272815bdc0 0x7f272806c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 -- 172.21.15.28:0/3376631035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272815ccf0 con 0x7f27280d08f0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272d000640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 0x7f2728155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f271fe00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100b1040 con 0x7f27280d08f0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100bc070 con 0x7f27280d08f0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272815e6d0 con 0x7f27280d08f0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100b7670 con 0x7f27280d08f0 2024-03-21T08:42:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272815ec90 con 0x7f27280d08f0 2024-03-21T08:42:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.608+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272809faf0 con 0x7f27280d08f0 2024-03-21T08:42:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.612+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f27100b7810 con 0x7f27280d08f0 2024-03-21T08:42:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.612+0000 7f2725600640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 0x7f2704069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.612+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2710120590 con 0x7f27280d08f0 2024-03-21T08:42:00.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.612+0000 7f272d000640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 0x7f2704069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.612+0000 7f272d000640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 0x7f2704069fe0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f27280cc150 tx=0x7f2718009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.616+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2710016090 con 0x7f27280d08f0 2024-03-21T08:42:00.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.780+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f272804f4f0 con 0x7f27280d08f0 2024-03-21T08:42:00.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.780+0000 7f2725600640 1 -- 172.21.15.28:0/3376631035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f27100f04a0 con 0x7f27280d08f0 2024-03-21T08:42:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:00.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 msgr2=0x7f2704069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 0x7f2704069fe0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f27280cc150 tx=0x7f2718009190 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 msgr2=0x7f272815c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001ac20 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 shutdown_connections 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f272815bdc0 0x7f272806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27280d08f0 0x7f272815c410 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2704067b20 0x7f2704069fe0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 --2- 172.21.15.28:0/3376631035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27280cc640 0x7f2728155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 >> 172.21.15.28:0/3376631035 conn(0x7f27280b3620 msgr2=0x7f272815a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 shutdown_connections 2024-03-21T08:42:00.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.784+0000 7f272fa00640 1 -- 172.21.15.28:0/3376631035 wait complete. 2024-03-21T08:42:00.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.944+0000 7f30ec600640 1 Processor -- start 2024-03-21T08:42:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.944+0000 7f30ec600640 1 -- start start 2024-03-21T08:42:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ec600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 0x7f30e40b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ec600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f30e4144690 0x7f30e40aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ec600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e40b2d20 con 0x7f30e40af120 2024-03-21T08:42:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ec600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e40b2ad0 con 0x7f30e4144690 2024-03-21T08:42:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ec600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40b3000 con 0x7f30e41442b0 2024-03-21T08:42:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ea600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f30e40af120 0x7f30e4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55550/0 (socket says 172.21.15.28:55550) 2024-03-21T08:42:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30ea600640 1 -- 172.21.15.28:0/2130375811 learned_addr learned my addr 172.21.15.28:0/2130375811 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9c00640 1 --2- 172.21.15.28:0/2130375811 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 0x7f30e40b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9200640 1 --2- 172.21.15.28:0/2130375811 >> v2:172.21.15.151:3300/0 conn(0x7f30e4144690 0x7f30e40aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e3e00640 1 -- 172.21.15.28:0/2130375811 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025263396 0 0) 0x7f30e40b2d20 con 0x7f30e40af120 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9c00640 1 -- 172.21.15.28:0/2130375811 >> v2:172.21.15.151:3300/0 conn(0x7f30e4144690 msgr2=0x7f30e40aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9c00640 1 --2- 172.21.15.28:0/2130375811 >> v2:172.21.15.151:3300/0 conn(0x7f30e4144690 0x7f30e40aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9c00640 1 -- 172.21.15.28:0/2130375811 >> v1:172.21.15.28:6789/0 conn(0x7f30e40af120 legacy=0x7f30e4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.948+0000 7f30e9c00640 1 -- 172.21.15.28:0/2130375811 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e40b32e0 con 0x7f30e41442b0 2024-03-21T08:42:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e9c00640 1 --2- 172.21.15.28:0/2130375811 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 0x7f30e40b3dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30d80097c0 tx=0x7f30d801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b00637d010823ab server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e3e00640 1 -- 172.21.15.28:0/2130375811 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80a7070 con 0x7f30e41442b0 2024-03-21T08:42:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e3e00640 1 -- 172.21.15.28:0/2130375811 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d80b0070 con 0x7f30e41442b0 2024-03-21T08:42:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e3e00640 1 -- 172.21.15.28:0/2130375811 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80ab890 con 0x7f30e41442b0 2024-03-21T08:42:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- 172.21.15.28:0/2130375811 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 msgr2=0x7f30e40b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- 172.21.15.28:0/2130375811 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 0x7f30e40b3dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f30d80097c0 tx=0x7f30d801c470 comp rx=0 tx=0).stop 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- 172.21.15.28:0/2130375811 shutdown_connections 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- 172.21.15.28:0/2130375811 >> v2:172.21.15.151:3300/0 conn(0x7f30e4144690 0x7f30e40aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- 172.21.15.28:0/2130375811 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e41442b0 0x7f30e40b3dc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- 172.21.15.28:0/2130375811 >> 172.21.15.28:0/2130375811 conn(0x7f30e40674f0 msgr2=0x7f30e406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- 172.21.15.28:0/2130375811 shutdown_connections 2024-03-21T08:42:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- 172.21.15.28:0/2130375811 wait complete. 2024-03-21T08:42:00.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 Processor -- start 2024-03-21T08:42:00.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- start start 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e40af120 0x7f30e415d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30e41442b0 0x7f30e4142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e406e2b0 con 0x7f30e4144690 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e40723a0 con 0x7f30e41442b0 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ec600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e406e430 con 0x7f30e40af120 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e9c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e40af120 0x7f30e415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53732/0 (socket says 172.21.15.28:53732) 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30ea600640 1 -- 172.21.15.28:0/3769911127 learned_addr learned my addr 172.21.15.28:0/3769911127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.952+0000 7f30e9200640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30e41442b0 0x7f30e4142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 -- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e40af120 msgr2=0x7f30e415d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e40af120 0x7f30e415d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 -- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30e41442b0 msgr2=0x7f30e4142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30e41442b0 0x7f30e4142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 -- 172.21.15.28:0/3769911127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4140a70 con 0x7f30e4144690 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ea600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00c7b0 tx=0x7f30dc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc018070 con 0x7f30e4144690 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc00a640 con 0x7f30e4144690 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00ce60 con 0x7f30e4144690 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e4162b50 con 0x7f30e4144690 2024-03-21T08:42:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e4163110 con 0x7f30e4144690 2024-03-21T08:42:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4073430 con 0x7f30e4144690 2024-03-21T08:42:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.956+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f30dc00aa30 con 0x7f30e4144690 2024-03-21T08:42:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.960+0000 7f30e1600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 0x7f30c406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.960+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f30dc0858d0 con 0x7f30e4144690 2024-03-21T08:42:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.960+0000 7f30e9c00640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 0x7f30c406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.960+0000 7f30e9c00640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 0x7f30c406a010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30d80af000 tx=0x7f30d80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:00.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:00.964+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f30e4140a70 con 0x7f30e4144690 2024-03-21T08:42:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.136+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f30e404f4f0 con 0x7f30e4144690 2024-03-21T08:42:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30e1600640 1 -- 172.21.15.28:0/3769911127 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f30dc0557e0 con 0x7f30e4144690 2024-03-21T08:42:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 msgr2=0x7f30c406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 0x7f30c406a010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30d80af000 tx=0x7f30d80a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 msgr2=0x7f30e41427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00c7b0 tx=0x7f30dc00cc70 comp rx=0 tx=0).stop 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 shutdown_connections 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e4144690 0x7f30e41427c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f30e41442b0 0x7f30e4142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f30c4067b50 0x7f30c406a010 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 --2- 172.21.15.28:0/3769911127 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f30e40af120 0x7f30e415d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 >> 172.21.15.28:0/3769911127 conn(0x7f30e40674f0 msgr2=0x7f30e4144a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 shutdown_connections 2024-03-21T08:42:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.140+0000 7f30ec600640 1 -- 172.21.15.28:0/3769911127 wait complete. 2024-03-21T08:42:01.163 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:42:01.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:01.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:01.184 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:01.184 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:42:01.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:01.184 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:42:01.236 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:01.236 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:01.246 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:01.246 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:01.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:01.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 Processor -- start 2024-03-21T08:42:01.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 -- start start 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 0x7f92c006cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00cc1d0 con 0x7f92c0099b80 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c006d450 con 0x7f92c0099fe0 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92c6800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00d1730 con 0x7f92c015e0c0 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92b7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f92c0099b80 0x7f92c00ccc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55552/0 (socket says 172.21.15.28:55552) 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92b7e00640 1 -- 172.21.15.28:0/1567076244 learned_addr learned my addr 172.21.15.28:0/1567076244 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bf400640 1 --2- 172.21.15.28:0/1567076244 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 0x7f92c006cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bea00640 1 -- 172.21.15.28:0/1567076244 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955487288 0 0) 0x7f92c00cc1d0 con 0x7f92c0099b80 2024-03-21T08:42:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bea00640 1 -- 172.21.15.28:0/1567076244 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a4003530 con 0x7f92c0099b80 2024-03-21T08:42:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bf400640 1 -- 172.21.15.28:0/1567076244 >> v1:172.21.15.151:6789/0 conn(0x7f92c0099fe0 legacy=0x7f92c00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bf400640 1 -- 172.21.15.28:0/1567076244 >> v1:172.21.15.28:6789/0 conn(0x7f92c0099b80 legacy=0x7f92c00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.440+0000 7f92bf400640 1 -- 172.21.15.28:0/1567076244 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00d1450 con 0x7f92c015e0c0 2024-03-21T08:42:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92bf400640 1 --2- 172.21.15.28:0/1567076244 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 0x7f92c006cf10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097f0 tx=0x7f92b001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349575e5ae897fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92bea00640 1 -- 172.21.15.28:0/1567076244 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b00b1070 con 0x7f92c015e0c0 2024-03-21T08:42:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92bea00640 1 -- 172.21.15.28:0/1567076244 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b00ac620 con 0x7f92c015e0c0 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92bea00640 1 -- 172.21.15.28:0/1567076244 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b00ac9d0 con 0x7f92c015e0c0 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 -- 172.21.15.28:0/1567076244 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 msgr2=0x7f92c006cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 --2- 172.21.15.28:0/1567076244 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 0x7f92c006cf10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097f0 tx=0x7f92b001c4b0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 -- 172.21.15.28:0/1567076244 shutdown_connections 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 --2- 172.21.15.28:0/1567076244 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c015e0c0 0x7f92c006cf10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 -- 172.21.15.28:0/1567076244 >> 172.21.15.28:0/1567076244 conn(0x7f92c00b34d0 msgr2=0x7f92c00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 -- 172.21.15.28:0/1567076244 shutdown_connections 2024-03-21T08:42:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.444+0000 7f92c6800640 1 -- 172.21.15.28:0/1567076244 wait complete. 2024-03-21T08:42:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 Processor -- start 2024-03-21T08:42:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- start start 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 0x7f92c0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92c0099fe0 0x7f92c00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00d10d0 con 0x7f92c015e0c0 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c009a3f0 con 0x7f92c0099fe0 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c016c4b0 con 0x7f92c0099b80 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92c0099fe0 0x7f92c00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53744/0 (socket says 172.21.15.28:53744) 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 -- 172.21.15.28:0/1445960871 learned_addr learned my addr 172.21.15.28:0/1445960871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bf400640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 0x7f92c0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 -- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 msgr2=0x7f92c0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 0x7f92c0160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 -- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92c0099fe0 msgr2=0x7f92c00b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92c0099fe0 0x7f92c00b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 -- 172.21.15.28:0/1445960871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00c6ee0 con 0x7f92c015e0c0 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bf400640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 0x7f92c0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c770 tx=0x7f92b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8003b80 con 0x7f92c015e0c0 2024-03-21T08:42:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b8003d20 con 0x7f92c015e0c0 2024-03-21T08:42:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b800aa90 con 0x7f92c015e0c0 2024-03-21T08:42:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00c71c0 con 0x7f92c015e0c0 2024-03-21T08:42:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c00c7a60 con 0x7f92c015e0c0 2024-03-21T08:42:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.448+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c00c7c60 con 0x7f92c015e0c0 2024-03-21T08:42:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.452+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f92b800ac30 con 0x7f92c015e0c0 2024-03-21T08:42:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.452+0000 7f92b7400640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 0x7f929c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.452+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f92b80860f0 con 0x7f92c015e0c0 2024-03-21T08:42:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.452+0000 7f92bf400640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 0x7f929c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.452+0000 7f92bf400640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 0x7f929c069fe0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f92b00ae000 tx=0x7f92b00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.456+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f92c00c7c60 con 0x7f92c015e0c0 2024-03-21T08:42:01.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c004f410 con 0x7f92c015e0c0 2024-03-21T08:42:01.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92b7400640 1 -- 172.21.15.28:0/1445960871 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f92b8056090 con 0x7f92c015e0c0 2024-03-21T08:42:01.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:01.622 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:01.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 msgr2=0x7f929c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 0x7f929c069fe0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f92b00ae000 tx=0x7f92b00a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 msgr2=0x7f92c0154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c770 tx=0x7f92b800cc30 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 shutdown_connections 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c015e0c0 0x7f92c0154a80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92c0099fe0 0x7f92c00b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f929c067b20 0x7f929c069fe0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 --2- 172.21.15.28:0/1445960871 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92c0099b80 0x7f92c0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 >> 172.21.15.28:0/1445960871 conn(0x7f92c00b34d0 msgr2=0x7f92c015d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 shutdown_connections 2024-03-21T08:42:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.620+0000 7f92c6800640 1 -- 172.21.15.28:0/1445960871 wait complete. 2024-03-21T08:42:01.643 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:01.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 Processor -- start 2024-03-21T08:42:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 -- start start 2024-03-21T08:42:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 0x7fcd3c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3c0d10f0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3c0d1380 con 0x7fcd3c0cca20 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd44800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c0d1660 con 0x7fcd3c0d0750 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd3c06cc90 0x7fcd3c06d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55562/0 (socket says 172.21.15.28:55562) 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41e00640 1 -- 172.21.15.28:0/2105401720 learned_addr learned my addr 172.21.15.28:0/2105401720 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd42800640 1 --2- 172.21.15.28:0/2105401720 >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 0x7fcd3c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41400640 1 --2- 172.21.15.28:0/2105401720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd40a00640 1 -- 172.21.15.28:0/2105401720 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151580495 0 0) 0x7fcd3c0d10f0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd40a00640 1 -- 172.21.15.28:0/2105401720 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd24003530 con 0x7fcd3c06cc90 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41400640 1 -- 172.21.15.28:0/2105401720 >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 msgr2=0x7fcd3c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41400640 1 --2- 172.21.15.28:0/2105401720 >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 0x7fcd3c0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41400640 1 -- 172.21.15.28:0/2105401720 >> v1:172.21.15.28:6789/0 conn(0x7fcd3c06cc90 legacy=0x7fcd3c06d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd41400640 1 -- 172.21.15.28:0/2105401720 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c0cc220 con 0x7fcd3c0d0750 2024-03-21T08:42:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.788+0000 7fcd42800640 1 --2- 172.21.15.28:0/2105401720 >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 0x7fcd3c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd41400640 1 --2- 172.21.15.28:0/2105401720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c0d0bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0097c0 tx=0x7fcd2c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2015d756760382 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd40a00640 1 -- 172.21.15.28:0/2105401720 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c0b0070 con 0x7fcd3c0d0750 2024-03-21T08:42:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd40a00640 1 -- 172.21.15.28:0/2105401720 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c0ab600 con 0x7fcd3c0d0750 2024-03-21T08:42:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd40a00640 1 -- 172.21.15.28:0/2105401720 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c0ab9b0 con 0x7fcd3c0d0750 2024-03-21T08:42:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 -- 172.21.15.28:0/2105401720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 msgr2=0x7fcd3c0d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 --2- 172.21.15.28:0/2105401720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c0d0bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0097c0 tx=0x7fcd2c01c410 comp rx=0 tx=0).stop 2024-03-21T08:42:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 -- 172.21.15.28:0/2105401720 shutdown_connections 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 --2- 172.21.15.28:0/2105401720 >> v2:172.21.15.151:3300/0 conn(0x7fcd3c0cca20 0x7fcd3c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 --2- 172.21.15.28:0/2105401720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c0d0bb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 -- 172.21.15.28:0/2105401720 >> 172.21.15.28:0/2105401720 conn(0x7fcd3c0b3540 msgr2=0x7fcd3c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 -- 172.21.15.28:0/2105401720 shutdown_connections 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.792+0000 7fcd44800640 1 -- 172.21.15.28:0/2105401720 wait complete. 2024-03-21T08:42:01.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 Processor -- start 2024-03-21T08:42:01.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- start start 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd3c0cca20 0x7fcd3c161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c16ada0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c16ac20 con 0x7fcd3c0cca20 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c16af20 con 0x7fcd3c0d0750 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53750/0 (socket says 172.21.15.28:53750) 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 -- 172.21.15.28:0/3032467290 learned_addr learned my addr 172.21.15.28:0/3032467290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41400640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd3c0cca20 0x7fcd3c161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd42800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 -- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 msgr2=0x7fcd3c1582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c1582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 -- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd3c0cca20 msgr2=0x7fcd3c161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd3c0cca20 0x7fcd3c161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 -- 172.21.15.28:0/3032467290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c15e660 con 0x7fcd3c06cc90 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c3c0 tx=0x7fcd3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30017070 con 0x7fcd3c06cc90 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd30005cf0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd300072f0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3c158800 con 0x7fcd3c06cc90 2024-03-21T08:42:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.796+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3c159160 con 0x7fcd3c06cc90 2024-03-21T08:42:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3c09f9c0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcd3001a070 con 0x7fcd3c06cc90 2024-03-21T08:42:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd36000640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 0x7fcd20069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fcd30085690 con 0x7fcd3c06cc90 2024-03-21T08:42:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd41400640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 0x7fcd20069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.800+0000 7fcd41400640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 0x7fcd20069fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0ad000 tx=0x7fcd2c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:01.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.804+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcd30055640 con 0x7fcd3c06cc90 2024-03-21T08:42:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.964+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd3c04f4f0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd36000640 1 -- 172.21.15.28:0/3032467290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd300596e0 con 0x7fcd3c06cc90 2024-03-21T08:42:01.969 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 msgr2=0x7fcd20069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 0x7fcd20069fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0ad000 tx=0x7fcd2c0a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 msgr2=0x7fcd3c158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c3c0 tx=0x7fcd3000c880 comp rx=0 tx=0).stop 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 shutdown_connections 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd20067b20 0x7fcd20069fe0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd3c0d0750 0x7fcd3c1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd3c0cca20 0x7fcd3c161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 --2- 172.21.15.28:0/3032467290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c06cc90 0x7fcd3c158b80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 >> 172.21.15.28:0/3032467290 conn(0x7fcd3c0b3540 msgr2=0x7fcd3c073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 shutdown_connections 2024-03-21T08:42:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:01.968+0000 7fcd44800640 1 -- 172.21.15.28:0/3032467290 wait complete. 2024-03-21T08:42:01.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 Processor -- start 2024-03-21T08:42:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 -- start start 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe020071a80 0x7fe02006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe0200cde40 0x7fe020071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0200723a0 con 0x7fe02015cbb0 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02015c720 con 0x7fe020071a80 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe028800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200ce5b0 con 0x7fe0200cde40 2024-03-21T08:42:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe020071a80 0x7fe02006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe026800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe02015cbb0 0x7fe0200cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55570/0 (socket says 172.21.15.28:55570) 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe026800640 1 -- 172.21.15.28:0/1763871574 learned_addr learned my addr 172.21.15.28:0/1763871574 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025400640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe0200cde40 0x7fe020071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025e00640 1 -- 172.21.15.28:0/1763871574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe0200cde40 msgr2=0x7fe020071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025e00640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe0200cde40 0x7fe020071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025e00640 1 -- 172.21.15.28:0/1763871574 >> v1:172.21.15.28:6789/0 conn(0x7fe02015cbb0 legacy=0x7fe0200cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe025e00640 1 -- 172.21.15.28:0/1763871574 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0200ce890 con 0x7fe020071a80 2024-03-21T08:42:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.136+0000 7fe024a00640 1 -- 172.21.15.28:0/1763871574 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394204543 0 0) 0x7fe0200723a0 con 0x7fe02015cbb0 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe025e00640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020071a80 0x7fe02006f0a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe0140097c0 tx=0x7fe01401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2276514e85bdd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe024a00640 1 -- 172.21.15.28:0/1763871574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140ab600 con 0x7fe020071a80 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe024a00640 1 -- 172.21.15.28:0/1763871574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0140b0070 con 0x7fe020071a80 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe024a00640 1 -- 172.21.15.28:0/1763871574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0140ab9c0 con 0x7fe020071a80 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 -- 172.21.15.28:0/1763871574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020071a80 msgr2=0x7fe02006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020071a80 0x7fe02006f0a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe0140097c0 tx=0x7fe01401c3e0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 -- 172.21.15.28:0/1763871574 shutdown_connections 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe0200cde40 0x7fe020071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 --2- 172.21.15.28:0/1763871574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020071a80 0x7fe02006f0a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 -- 172.21.15.28:0/1763871574 >> 172.21.15.28:0/1763871574 conn(0x7fe0200674f0 msgr2=0x7fe02006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 -- 172.21.15.28:0/1763871574 shutdown_connections 2024-03-21T08:42:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.140+0000 7fe028800640 1 -- 172.21.15.28:0/1763871574 wait complete. 2024-03-21T08:42:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 Processor -- start 2024-03-21T08:42:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- start start 2024-03-21T08:42:02.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe020071a80 0x7fe0200a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0200cde40 0x7fe0200b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe020168f20 con 0x7fe02015cbb0 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200ce220 con 0x7fe0200cde40 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0201690a0 con 0x7fe020071a80 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53776/0 (socket says 172.21.15.28:53776) 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 -- 172.21.15.28:0/964861098 learned_addr learned my addr 172.21.15.28:0/964861098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe025400640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0200cde40 0x7fe0200b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe025e00640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe020071a80 0x7fe0200a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 -- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe020071a80 msgr2=0x7fe0200a8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe020071a80 0x7fe0200a8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 -- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0200cde40 msgr2=0x7fe0200b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0200cde40 0x7fe0200b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 -- 172.21.15.28:0/964861098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02013df90 con 0x7fe02015cbb0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe026800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c7b0 tx=0x7fe01c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c018070 con 0x7fe02015cbb0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c004d70 con 0x7fe02015cbb0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c00a880 con 0x7fe02015cbb0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0200a8ff0 con 0x7fe02015cbb0 2024-03-21T08:42:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.144+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0200a95b0 con 0x7fe02015cbb0 2024-03-21T08:42:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020073430 con 0x7fe02015cbb0 2024-03-21T08:42:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe01c00aa20 con 0x7fe02015cbb0 2024-03-21T08:42:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe01a000640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 0x7fe004069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fe01c0858c0 con 0x7fe02015cbb0 2024-03-21T08:42:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe025e00640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 0x7fe004069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.148+0000 7fe025e00640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 0x7fe004069fe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe0140af000 tx=0x7fe0140a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.152+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe01c055870 con 0x7fe02015cbb0 2024-03-21T08:42:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.316+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0200afea0 con 0x7fe02015cbb0 2024-03-21T08:42:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.316+0000 7fe01a000640 1 -- 172.21.15.28:0/964861098 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe01c059910 con 0x7fe02015cbb0 2024-03-21T08:42:02.320 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 msgr2=0x7fe004069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 0x7fe004069fe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe0140af000 tx=0x7fe0140a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 msgr2=0x7fe02013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00c7b0 tx=0x7fe01c00cc70 comp rx=0 tx=0).stop 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 shutdown_connections 2024-03-21T08:42:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe02015cbb0 0x7fe02013be10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe0200cde40 0x7fe0200b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe004067b20 0x7fe004069fe0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 --2- 172.21.15.28:0/964861098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe020071a80 0x7fe0200a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 >> 172.21.15.28:0/964861098 conn(0x7fe0200674f0 msgr2=0x7fe0200727c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 shutdown_connections 2024-03-21T08:42:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.320+0000 7fe028800640 1 -- 172.21.15.28:0/964861098 wait complete. 2024-03-21T08:42:02.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:02.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 Processor -- start 2024-03-21T08:42:02.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 -- start start 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ee8071a80 0x7f9ee806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9ee815cbb0 0x7f9ee80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee80ce5b0 con 0x7f9ee80cde40 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee815c720 con 0x7f9ee8071a80 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9eed600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee80ce890 con 0x7f9ee815cbb0 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ee8071a80 0x7f9ee806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9ee80cde40 0x7f9ee8071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55580/0 (socket says 172.21.15.28:55580) 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6000640 1 -- 172.21.15.28:0/3702443091 learned_addr learned my addr 172.21.15.28:0/3702443091 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee7400640 1 --2- 172.21.15.28:0/3702443091 >> v2:172.21.15.151:3300/0 conn(0x7f9ee815cbb0 0x7f9ee80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6a00640 1 -- 172.21.15.28:0/3702443091 >> v2:172.21.15.151:3300/0 conn(0x7f9ee815cbb0 msgr2=0x7f9ee80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6a00640 1 --2- 172.21.15.28:0/3702443091 >> v2:172.21.15.151:3300/0 conn(0x7f9ee815cbb0 0x7f9ee80cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6a00640 1 -- 172.21.15.28:0/3702443091 >> v1:172.21.15.28:6789/0 conn(0x7f9ee80cde40 legacy=0x7f9ee8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee6a00640 1 -- 172.21.15.28:0/3702443091 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee815c3b0 con 0x7f9ee8071a80 2024-03-21T08:42:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.484+0000 7f9ee5600640 1 -- 172.21.15.28:0/3702443091 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357228538 0 0) 0x7f9ee80ce5b0 con 0x7f9ee80cde40 2024-03-21T08:42:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9ee6a00640 1 --2- 172.21.15.28:0/3702443091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee806f0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80b5030 tx=0x7f9ed801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fef562914484209 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9ee5600640 1 -- 172.21.15.28:0/3702443091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80ab610 con 0x7f9ee8071a80 2024-03-21T08:42:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9ee5600640 1 -- 172.21.15.28:0/3702443091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed80b0070 con 0x7f9ee8071a80 2024-03-21T08:42:02.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9ee5600640 1 -- 172.21.15.28:0/3702443091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80ab9d0 con 0x7f9ee8071a80 2024-03-21T08:42:02.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- 172.21.15.28:0/3702443091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 msgr2=0x7f9ee806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 --2- 172.21.15.28:0/3702443091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee806f0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80b5030 tx=0x7f9ed801c3f0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- 172.21.15.28:0/3702443091 shutdown_connections 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 --2- 172.21.15.28:0/3702443091 >> v2:172.21.15.151:3300/0 conn(0x7f9ee815cbb0 0x7f9ee80cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 --2- 172.21.15.28:0/3702443091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee806f0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- 172.21.15.28:0/3702443091 >> 172.21.15.28:0/3702443091 conn(0x7f9ee80674f0 msgr2=0x7f9ee806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- 172.21.15.28:0/3702443091 shutdown_connections 2024-03-21T08:42:02.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- 172.21.15.28:0/3702443091 wait complete. 2024-03-21T08:42:02.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 Processor -- start 2024-03-21T08:42:02.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.488+0000 7f9eed600640 1 -- start start 2024-03-21T08:42:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ee80cde40 0x7f9ee815afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ee815cbb0 0x7f9ee8142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8168f20 con 0x7f9ee8071a80 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8144b20 con 0x7f9ee815cbb0 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee81690a0 con 0x7f9ee80cde40 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53802/0 (socket says 172.21.15.28:53802) 2024-03-21T08:42:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 -- 172.21.15.28:0/2421881612 learned_addr learned my addr 172.21.15.28:0/2421881612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee7400640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ee815cbb0 0x7f9ee8142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6000640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ee80cde40 0x7f9ee815afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 -- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ee80cde40 msgr2=0x7f9ee815afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ee80cde40 0x7f9ee815afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 -- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ee815cbb0 msgr2=0x7f9ee8142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ee815cbb0 0x7f9ee8142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 -- 172.21.15.28:0/2421881612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee815b500 con 0x7f9ee8071a80 2024-03-21T08:42:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9ee6a00640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80ad040 tx=0x7f9ed801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80bf410 con 0x7f9ee8071a80 2024-03-21T08:42:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed80bf5b0 con 0x7f9ee8071a80 2024-03-21T08:42:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80bf9a0 con 0x7f9ee8071a80 2024-03-21T08:42:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee8143810 con 0x7f9ee8071a80 2024-03-21T08:42:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee80afac0 con 0x7f9ee8071a80 2024-03-21T08:42:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.492+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee804f4f0 con 0x7f9ee8071a80 2024-03-21T08:42:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.496+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9ed80bd020 con 0x7f9ee8071a80 2024-03-21T08:42:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.496+0000 7f9edea00640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 0x7f9ec8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.496+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f9ed81202c0 con 0x7f9ee8071a80 2024-03-21T08:42:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.496+0000 7f9ee6000640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 0x7f9ec8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.496+0000 7f9ee6000640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 0x7f9ec8069fe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8143c70 tx=0x7f9ed4007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.500+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9ed80c6070 con 0x7f9ee8071a80 2024-03-21T08:42:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ee8073410 con 0x7f9ee8071a80 2024-03-21T08:42:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9edea00640 1 -- 172.21.15.28:0/2421881612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ed80a8050 con 0x7f9ee8071a80 2024-03-21T08:42:02.665 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 msgr2=0x7f9ec8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 0x7f9ec8069fe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8143c70 tx=0x7f9ed4007d60 comp rx=0 tx=0).stop 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 msgr2=0x7f9ee8133ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80ad040 tx=0x7f9ed801ac30 comp rx=0 tx=0).stop 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 shutdown_connections 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ee815cbb0 0x7f9ee8142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9ec8067b20 0x7f9ec8069fe0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ee80cde40 0x7f9ee815afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 --2- 172.21.15.28:0/2421881612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8071a80 0x7f9ee8133ff0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 >> 172.21.15.28:0/2421881612 conn(0x7f9ee80674f0 msgr2=0x7f9ee80b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 shutdown_connections 2024-03-21T08:42:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.664+0000 7f9eed600640 1 -- 172.21.15.28:0/2421881612 wait complete. 2024-03-21T08:42:02.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:02.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.828+0000 7f9e2ea00640 1 Processor -- start 2024-03-21T08:42:02.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.828+0000 7f9e2ea00640 1 -- start start 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 0x7f9e280cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2ea00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2ea00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280d03e0 con 0x7f9e280cff70 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2ea00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280cc340 con 0x7f9e28071c90 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2ea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280ce3d0 con 0x7f9e2815dba0 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2cc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2d600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 0x7f9e280cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2e000640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f9e280cff70 0x7f9e28161810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:48922/0 (socket says 172.21.15.28:48922) 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2cc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35492/0 (socket says 172.21.15.28:35492) 2024-03-21T08:42:02.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2e000640 1 -- 172.21.15.28:0/3441704019 learned_addr learned my addr 172.21.15.28:0/3441704019 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e27e00640 1 -- 172.21.15.28:0/3441704019 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126315293 0 0) 0x7f9e280d03e0 con 0x7f9e280cff70 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2d600640 1 -- 172.21.15.28:0/3441704019 >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 msgr2=0x7f9e280cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2d600640 1 --2- 172.21.15.28:0/3441704019 >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2d600640 1 -- 172.21.15.28:0/3441704019 >> v1:172.21.15.151:6789/0 conn(0x7f9e280cff70 legacy=0x7f9e28161810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2d600640 1 -- 172.21.15.28:0/3441704019 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280ce6b0 con 0x7f9e2815dba0 2024-03-21T08:42:02.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.832+0000 7f9e2cc00640 1 --2- 172.21.15.28:0/3441704019 >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:02.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2d600640 1 --2- 172.21.15.28:0/3441704019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 0x7f9e280cddf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0097c0 tx=0x7f9e1c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cddd2aebaee80ca server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e27e00640 1 -- 172.21.15.28:0/3441704019 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0b2070 con 0x7f9e2815dba0 2024-03-21T08:42:02.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e27e00640 1 -- 172.21.15.28:0/3441704019 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c0ad640 con 0x7f9e2815dba0 2024-03-21T08:42:02.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e27e00640 1 -- 172.21.15.28:0/3441704019 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0ada10 con 0x7f9e2815dba0 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- 172.21.15.28:0/3441704019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 msgr2=0x7f9e280cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/3441704019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 0x7f9e280cddf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0097c0 tx=0x7f9e1c01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- 172.21.15.28:0/3441704019 shutdown_connections 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/3441704019 >> v2:172.21.15.151:3300/0 conn(0x7f9e28071c90 0x7f9e280cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/3441704019 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e2815dba0 0x7f9e280cddf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- 172.21.15.28:0/3441704019 >> 172.21.15.28:0/3441704019 conn(0x7f9e2806fa90 msgr2=0x7f9e2806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- 172.21.15.28:0/3441704019 shutdown_connections 2024-03-21T08:42:02.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- 172.21.15.28:0/3441704019 wait complete. 2024-03-21T08:42:02.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 Processor -- start 2024-03-21T08:42:02.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- start start 2024-03-21T08:42:02.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e28071c90 0x7f9e28154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e2815dba0 0x7f9e2806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815e100 con 0x7f9e280cff70 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815df80 con 0x7f9e2815dba0 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.836+0000 7f9e2ea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2816ab90 con 0x7f9e28071c90 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53810/0 (socket says 172.21.15.28:53810) 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 -- 172.21.15.28:0/2430077856 learned_addr learned my addr 172.21.15.28:0/2430077856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2d600640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e28071c90 0x7f9e28154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2e000640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e2815dba0 0x7f9e2806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 -- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e28071c90 msgr2=0x7f9e28154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e28071c90 0x7f9e28154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 -- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e2815dba0 msgr2=0x7f9e2806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e2815dba0 0x7f9e2806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 -- 172.21.15.28:0/2430077856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280c54c0 con 0x7f9e280cff70 2024-03-21T08:42:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2cc00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400ebe0 tx=0x7f9e1400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14017070 con 0x7f9e280cff70 2024-03-21T08:42:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400cc30 con 0x7f9e280cff70 2024-03-21T08:42:02.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14013470 con 0x7f9e280cff70 2024-03-21T08:42:02.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e280c57a0 con 0x7f9e280cff70 2024-03-21T08:42:02.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2815e5f0 con 0x7f9e280cff70 2024-03-21T08:42:02.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2804f410 con 0x7f9e280cff70 2024-03-21T08:42:02.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9e1401a070 con 0x7f9e280cff70 2024-03-21T08:42:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.840+0000 7f9e25600640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 0x7f9e08069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.844+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f9e14086640 con 0x7f9e280cff70 2024-03-21T08:42:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.844+0000 7f9e2d600640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 0x7f9e08069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.844+0000 7f9e2d600640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 0x7f9e08069fe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0af000 tx=0x7f9e1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:02.844+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9e14056550 con 0x7f9e280cff70 2024-03-21T08:42:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.008+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9e28071520 con 0x7f9e280cff70 2024-03-21T08:42:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.008+0000 7f9e25600640 1 -- 172.21.15.28:0/2430077856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9e1405a5f0 con 0x7f9e280cff70 2024-03-21T08:42:03.011 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 msgr2=0x7f9e08069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 0x7f9e08069fe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0af000 tx=0x7f9e1c008040 comp rx=0 tx=0).stop 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 msgr2=0x7f9e280c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400ebe0 tx=0x7f9e1400c520 comp rx=0 tx=0).stop 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 shutdown_connections 2024-03-21T08:42:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9e2815dba0 0x7f9e2806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e280cff70 0x7f9e280c7af0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e08067b20 0x7f9e08069fe0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 --2- 172.21.15.28:0/2430077856 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9e28071c90 0x7f9e28154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 >> 172.21.15.28:0/2430077856 conn(0x7f9e2806fa90 msgr2=0x7f9e280d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 shutdown_connections 2024-03-21T08:42:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.012+0000 7f9e2ea00640 1 -- 172.21.15.28:0/2430077856 wait complete. 2024-03-21T08:42:03.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 Processor -- start 2024-03-21T08:42:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 -- start start 2024-03-21T08:42:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff4500b4800 0x7ff4500ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff450072190 0x7ff4500cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4500cd670 con 0x7ff4500b4420 2024-03-21T08:42:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45015bf90 con 0x7ff450072190 2024-03-21T08:42:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff456a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45015b1d0 con 0x7ff4500b4800 2024-03-21T08:42:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff454a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff450072190 0x7ff4500cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff44fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4500b4420 0x7ff450071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55594/0 (socket says 172.21.15.28:55594) 2024-03-21T08:42:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff44fe00640 1 -- 172.21.15.28:0/3345251619 learned_addr learned my addr 172.21.15.28:0/3345251619 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff447400640 1 --2- 172.21.15.28:0/3345251619 >> v2:172.21.15.151:3300/0 conn(0x7ff4500b4800 0x7ff4500ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff454a00640 1 -- 172.21.15.28:0/3345251619 >> v2:172.21.15.151:3300/0 conn(0x7ff4500b4800 msgr2=0x7ff4500ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff454a00640 1 --2- 172.21.15.28:0/3345251619 >> v2:172.21.15.151:3300/0 conn(0x7ff4500b4800 0x7ff4500ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff454a00640 1 -- 172.21.15.28:0/3345251619 >> v1:172.21.15.28:6789/0 conn(0x7ff4500b4420 legacy=0x7ff450071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.176+0000 7ff454a00640 1 -- 172.21.15.28:0/3345251619 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45015b4b0 con 0x7ff450072190 2024-03-21T08:42:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff454a00640 1 --2- 172.21.15.28:0/3345251619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff4500cd130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff4500cc6f0 tx=0x7ff44801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5911db43e29c1b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff44f400640 1 -- 172.21.15.28:0/3345251619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4480b2070 con 0x7ff450072190 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff44f400640 1 -- 172.21.15.28:0/3345251619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4480a59c0 con 0x7ff450072190 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff44f400640 1 -- 172.21.15.28:0/3345251619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4480a5d90 con 0x7ff450072190 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- 172.21.15.28:0/3345251619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 msgr2=0x7ff4500cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 --2- 172.21.15.28:0/3345251619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff4500cd130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff4500cc6f0 tx=0x7ff44801c900 comp rx=0 tx=0).stop 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- 172.21.15.28:0/3345251619 shutdown_connections 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 --2- 172.21.15.28:0/3345251619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff4500cd130 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 --2- 172.21.15.28:0/3345251619 >> v2:172.21.15.151:3300/0 conn(0x7ff4500b4800 0x7ff4500ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- 172.21.15.28:0/3345251619 >> 172.21.15.28:0/3345251619 conn(0x7ff4500674f0 msgr2=0x7ff45006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- 172.21.15.28:0/3345251619 shutdown_connections 2024-03-21T08:42:03.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- 172.21.15.28:0/3345251619 wait complete. 2024-03-21T08:42:03.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 Processor -- start 2024-03-21T08:42:03.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.180+0000 7ff456a00640 1 -- start start 2024-03-21T08:42:03.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4500b4420 0x7ff4500c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4500c5890 0x7ff450150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b3460 con 0x7ff450072190 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b32e0 con 0x7ff4500c5890 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b35e0 con 0x7ff4500b4420 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53834/0 (socket says 172.21.15.28:53834) 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 -- 172.21.15.28:0/758295721 learned_addr learned my addr 172.21.15.28:0/758295721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff454a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4500c5890 0x7ff450150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff447400640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4500b4420 0x7ff4500c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 -- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4500b4420 msgr2=0x7ff4500c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4500b4420 0x7ff4500c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 -- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4500c5890 msgr2=0x7ff450150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4500c5890 0x7ff450150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 -- 172.21.15.28:0/758295721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4501512e0 con 0x7ff450072190 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44fe00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff44000c420 tx=0x7ff44000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440017070 con 0x7ff450072190 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff440005cf0 con 0x7ff450072190 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400072f0 con 0x7ff450072190 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff450148320 con 0x7ff450072190 2024-03-21T08:42:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4501488e0 con 0x7ff450072190 2024-03-21T08:42:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.184+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4500c5ca0 con 0x7ff450072190 2024-03-21T08:42:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.188+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff44001a070 con 0x7ff450072190 2024-03-21T08:42:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.188+0000 7ff44cc00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 0x7ff42c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.188+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7ff4400856e0 con 0x7ff450072190 2024-03-21T08:42:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.188+0000 7ff447400640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 0x7ff42c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.188+0000 7ff447400640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 0x7ff42c069fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff450151df0 tx=0x7ff438009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.192+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff4500c5ca0 con 0x7ff450072190 2024-03-21T08:42:03.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff450073430 con 0x7ff450072190 2024-03-21T08:42:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff44cc00640 1 -- 172.21.15.28:0/758295721 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff4400555f0 con 0x7ff450072190 2024-03-21T08:42:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 msgr2=0x7ff42c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 0x7ff42c069fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff450151df0 tx=0x7ff438009190 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 msgr2=0x7ff450147790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff44000c420 tx=0x7ff44000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 shutdown_connections 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff4500c5890 0x7ff450150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff42c067b20 0x7ff42c069fe0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff4500b4420 0x7ff4500c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 --2- 172.21.15.28:0/758295721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff450072190 0x7ff450147790 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 >> 172.21.15.28:0/758295721 conn(0x7ff4500674f0 msgr2=0x7ff45015c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 shutdown_connections 2024-03-21T08:42:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.368+0000 7ff456a00640 1 -- 172.21.15.28:0/758295721 wait complete. 2024-03-21T08:42:03.390 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:03.390 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:03.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 Processor -- start 2024-03-21T08:42:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 -- start start 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0c0cdb50 con 0x7f8b0c0cd5d0 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0c0c5530 con 0x7f8b0c09a290 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b14800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0cde30 con 0x7f8b0c15f260 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b12800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b12800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35518/0 (socket says 172.21.15.28:35518) 2024-03-21T08:42:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b12800640 1 -- 172.21.15.28:0/1990964747 learned_addr learned my addr 172.21.15.28:0/1990964747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11400640 1 --1- 172.21.15.28:0/1990964747 >> v1:172.21.15.151:6789/0 conn(0x7f8b0c0cd5d0 0x7f8b0c099b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:48932/0 (socket says 172.21.15.28:48932) 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11e00640 1 -- 172.21.15.28:0/1990964747 >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 msgr2=0x7f8b0c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11e00640 1 --2- 172.21.15.28:0/1990964747 >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11e00640 1 -- 172.21.15.28:0/1990964747 >> v1:172.21.15.151:6789/0 conn(0x7f8b0c0cd5d0 legacy=0x7f8b0c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b11e00640 1 -- 172.21.15.28:0/1990964747 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c0c5780 con 0x7f8b0c15f260 2024-03-21T08:42:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.540+0000 7f8b12800640 1 --2- 172.21.15.28:0/1990964747 >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b11e00640 1 --2- 172.21.15.28:0/1990964747 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c0cd090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b000097c0 tx=0x7f8b0001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e81ee370e91b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b10a00640 1 -- 172.21.15.28:0/1990964747 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000ab600 con 0x7f8b0c15f260 2024-03-21T08:42:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b10a00640 1 -- 172.21.15.28:0/1990964747 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b000b0070 con 0x7f8b0c15f260 2024-03-21T08:42:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b10a00640 1 -- 172.21.15.28:0/1990964747 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b000ab9c0 con 0x7f8b0c15f260 2024-03-21T08:42:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 -- 172.21.15.28:0/1990964747 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 msgr2=0x7f8b0c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 --2- 172.21.15.28:0/1990964747 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c0cd090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b000097c0 tx=0x7f8b0001c3e0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 -- 172.21.15.28:0/1990964747 shutdown_connections 2024-03-21T08:42:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 --2- 172.21.15.28:0/1990964747 >> v2:172.21.15.151:3300/0 conn(0x7f8b0c09a290 0x7f8b0c0c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 --2- 172.21.15.28:0/1990964747 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c0cd090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 -- 172.21.15.28:0/1990964747 >> 172.21.15.28:0/1990964747 conn(0x7f8b0c0b34d0 msgr2=0x7f8b0c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 -- 172.21.15.28:0/1990964747 shutdown_connections 2024-03-21T08:42:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.544+0000 7f8b14800640 1 -- 172.21.15.28:0/1990964747 wait complete. 2024-03-21T08:42:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 Processor -- start 2024-03-21T08:42:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- start start 2024-03-21T08:42:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b0c09a290 0x7f8b0c156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c09cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0c6e50 con 0x7f8b0c0cd5d0 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0c6cd0 con 0x7f8b0c09a290 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0c6fd0 con 0x7f8b0c15f260 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53844/0 (socket says 172.21.15.28:53844) 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c09cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b11400640 1 -- 172.21.15.28:0/1770018881 learned_addr learned my addr 172.21.15.28:0/1770018881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b11e00640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b0c09a290 0x7f8b0c156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 -- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b0c09a290 msgr2=0x7f8b0c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b0c09a290 0x7f8b0c156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 -- 172.21.15.28:0/1770018881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 msgr2=0x7f8b0c155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 -- 172.21.15.28:0/1770018881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c06c400 con 0x7f8b0c15f260 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b11400640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b12800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c09cc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800efc0 tx=0x7f8b0800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800cac0 con 0x7f8b0c15f260 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0800cc80 con 0x7f8b0c15f260 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08003be0 con 0x7f8b0c15f260 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c06c6e0 con 0x7f8b0c15f260 2024-03-21T08:42:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c06cca0 con 0x7f8b0c15f260 2024-03-21T08:42:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.548+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c04f410 con 0x7f8b0c15f260 2024-03-21T08:42:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.552+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8b0801b030 con 0x7f8b0c15f260 2024-03-21T08:42:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.552+0000 7f8b06000640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 0x7f8af0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.552+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f8b08086500 con 0x7f8b0c15f260 2024-03-21T08:42:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.552+0000 7f8b11e00640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 0x7f8af0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.552+0000 7f8b11e00640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 0x7f8af0069fe0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b000af000 tx=0x7f8b000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.556+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8b08022020 con 0x7f8b0c15f260 2024-03-21T08:42:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b0c0b2f90 con 0x7f8b0c15f260 2024-03-21T08:42:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b06000640 1 -- 172.21.15.28:0/1770018881 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f8b08010040 con 0x7f8b0c15f260 2024-03-21T08:42:03.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:03.726 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 msgr2=0x7f8af0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 0x7f8af0069fe0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b000af000 tx=0x7f8b000a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 msgr2=0x7f8b0c09cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.724+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c09cc20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800efc0 tx=0x7f8b0800c490 comp rx=0 tx=0).stop 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 shutdown_connections 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8af0067b20 0x7f8af0069fe0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8b0c15f260 0x7f8b0c09cc20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0cd5d0 0x7f8b0c155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 --2- 172.21.15.28:0/1770018881 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8b0c09a290 0x7f8b0c156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 >> 172.21.15.28:0/1770018881 conn(0x7f8b0c0b34d0 msgr2=0x7f8b0c0cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 shutdown_connections 2024-03-21T08:42:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.728+0000 7f8b14800640 1 -- 172.21.15.28:0/1770018881 wait complete. 2024-03-21T08:42:03.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:03.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 Processor -- start 2024-03-21T08:42:03.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 -- start start 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad840b2de0 0x7fad8415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad84072310 0x7fad840727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad840cee20 con 0x7fad840aedd0 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840cbdf0 con 0x7fad84072310 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad8ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad840cf100 con 0x7fad840b2de0 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad84072310 0x7fad840727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad84072310 0x7fad840727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53852/0 (socket says 172.21.15.28:53852) 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 -- 172.21.15.28:0/4193510639 learned_addr learned my addr 172.21.15.28:0/4193510639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:03.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad83400640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad840b2de0 0x7fad8415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 -- 172.21.15.28:0/4193510639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad840b2de0 msgr2=0x7fad8415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:03.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad840b2de0 0x7fad8415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 -- 172.21.15.28:0/4193510639 >> v1:172.21.15.151:6789/0 conn(0x7fad840aedd0 legacy=0x7fad840ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.892+0000 7fad88e00640 1 -- 172.21.15.28:0/4193510639 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad840cf3e0 con 0x7fad84072310 2024-03-21T08:42:03.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad88e00640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad84072310 0x7fad840727a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009e50 tx=0x7fad7c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebce32672d690ac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad82a00640 1 -- 172.21.15.28:0/4193510639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0a5680 con 0x7fad84072310 2024-03-21T08:42:03.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad82a00640 1 -- 172.21.15.28:0/4193510639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0a5820 con 0x7fad84072310 2024-03-21T08:42:03.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad82a00640 1 -- 172.21.15.28:0/4193510639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0a5bf0 con 0x7fad84072310 2024-03-21T08:42:03.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- 172.21.15.28:0/4193510639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad84072310 msgr2=0x7fad840727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad84072310 0x7fad840727a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009e50 tx=0x7fad7c01ca30 comp rx=0 tx=0).stop 2024-03-21T08:42:03.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- 172.21.15.28:0/4193510639 shutdown_connections 2024-03-21T08:42:03.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad84072310 0x7fad840727a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 --2- 172.21.15.28:0/4193510639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad840b2de0 0x7fad8415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- 172.21.15.28:0/4193510639 >> 172.21.15.28:0/4193510639 conn(0x7fad840674f0 msgr2=0x7fad8406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- 172.21.15.28:0/4193510639 shutdown_connections 2024-03-21T08:42:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- 172.21.15.28:0/4193510639 wait complete. 2024-03-21T08:42:03.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 Processor -- start 2024-03-21T08:42:03.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.896+0000 7fad8ae00640 1 -- start start 2024-03-21T08:42:03.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 0x7fad841f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 0x7fad841597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840cc630 con 0x7fad840b2de0 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840cc4b0 con 0x7fad840aedd0 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad840cc7b0 con 0x7fad84072310 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 0x7fad841f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 0x7fad841597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 0x7fad841f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:37758/0 (socket says 172.21.15.28:37758) 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83e00640 1 -- 172.21.15.28:0/2621341849 learned_addr learned my addr 172.21.15.28:0/2621341849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35536/0 (socket says 172.21.15.28:35536) 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 -- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 msgr2=0x7fad841f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 0x7fad841f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 -- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 msgr2=0x7fad84141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 -- 172.21.15.28:0/2621341849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad841f79e0 con 0x7fad840b2de0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad83400640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad88e00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 0x7fad841597c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0ab000 tx=0x7fad7c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0bc030 con 0x7fad840b2de0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0b8470 con 0x7fad840b2de0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad840b0ee0 con 0x7fad840b2de0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c009510 con 0x7fad840b2de0 2024-03-21T08:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad840b14a0 con 0x7fad840b2de0 2024-03-21T08:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.900+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84073430 con 0x7fad840b2de0 2024-03-21T08:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.904+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fad7c0bf070 con 0x7fad840b2de0 2024-03-21T08:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.904+0000 7fad7be00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 0x7fad64069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.904+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fad7c120b40 con 0x7fad840b2de0 2024-03-21T08:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.904+0000 7fad83e00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 0x7fad64069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.904+0000 7fad83e00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 0x7fad64069fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fad841f4610 tx=0x7fad74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:03.908+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fad841f79e0 con 0x7fad840b2de0 2024-03-21T08:42:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad8404f4f0 con 0x7fad840b2de0 2024-03-21T08:42:04.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad7be00640 1 -- 172.21.15.28:0/2621341849 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad7c016620 con 0x7fad840b2de0 2024-03-21T08:42:04.077 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 msgr2=0x7fad64069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 0x7fad64069fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fad841f4610 tx=0x7fad74009040 comp rx=0 tx=0).stop 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 msgr2=0x7fad841597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 0x7fad841597c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0ab000 tx=0x7fad7c0197d0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 shutdown_connections 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad840b2de0 0x7fad841597c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad840aedd0 0x7fad84141710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad64067b20 0x7fad64069fe0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 --2- 172.21.15.28:0/2621341849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad84072310 0x7fad841f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 >> 172.21.15.28:0/2621341849 conn(0x7fad840674f0 msgr2=0x7fad841632f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 shutdown_connections 2024-03-21T08:42:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.076+0000 7fad8ae00640 1 -- 172.21.15.28:0/2621341849 wait complete. 2024-03-21T08:42:04.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.240+0000 7f5665800640 1 Processor -- start 2024-03-21T08:42:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.240+0000 7f5665800640 1 -- start start 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f5665800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56600cde40 0x7f56600cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f5665800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566006f500 con 0x7f5660071a80 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f5665800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600ce5b0 con 0x7f566006f0a0 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f5665800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566015c720 con 0x7f56600cde40 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56600cde40 0x7f56600cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5660071a80 0x7f5660071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55610/0 (socket says 172.21.15.28:55610) 2024-03-21T08:42:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565ea00640 1 -- 172.21.15.28:0/1769247603 learned_addr learned my addr 172.21.15.28:0/1769247603 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565f400640 1 -- 172.21.15.28:0/1769247603 >> v1:172.21.15.151:6789/0 conn(0x7f566006f0a0 legacy=0x7f566015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565f400640 1 -- 172.21.15.28:0/1769247603 >> v1:172.21.15.28:6789/0 conn(0x7f5660071a80 legacy=0x7f5660071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.244+0000 7f565f400640 1 -- 172.21.15.28:0/1769247603 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56600ce890 con 0x7f56600cde40 2024-03-21T08:42:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f565f400640 1 --2- 172.21.15.28:0/1769247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56600cde40 0x7f56600cc4b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5658009d90 tx=0x7f565801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aff9ed77c2e8aea server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f565e000640 1 -- 172.21.15.28:0/1769247603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580b2070 con 0x7f56600cde40 2024-03-21T08:42:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f565e000640 1 -- 172.21.15.28:0/1769247603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56580a58b0 con 0x7f56600cde40 2024-03-21T08:42:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f565e000640 1 -- 172.21.15.28:0/1769247603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580a5c80 con 0x7f56600cde40 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- 172.21.15.28:0/1769247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56600cde40 msgr2=0x7f56600cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 --2- 172.21.15.28:0/1769247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56600cde40 0x7f56600cc4b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5658009d90 tx=0x7f565801c8e0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- 172.21.15.28:0/1769247603 shutdown_connections 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 --2- 172.21.15.28:0/1769247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56600cde40 0x7f56600cc4b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- 172.21.15.28:0/1769247603 >> 172.21.15.28:0/1769247603 conn(0x7f56600674f0 msgr2=0x7f566006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- 172.21.15.28:0/1769247603 shutdown_connections 2024-03-21T08:42:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- 172.21.15.28:0/1769247603 wait complete. 2024-03-21T08:42:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 Processor -- start 2024-03-21T08:42:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 -- start start 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.248+0000 7f5665800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5660071a80 0x7f56601414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56600cde40 0x7f5660141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5660144b20 con 0x7f566006f0a0 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600cce00 con 0x7f5660071a80 2024-03-21T08:42:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5660169f30 con 0x7f56600cde40 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53870/0 (socket says 172.21.15.28:53870) 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 -- 172.21.15.28:0/3283783229 learned_addr learned my addr 172.21.15.28:0/3283783229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5657e00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5660071a80 0x7f56601414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565f400640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56600cde40 0x7f5660141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 -- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56600cde40 msgr2=0x7f5660141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56600cde40 0x7f5660141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 -- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5660071a80 msgr2=0x7f56601414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5660071a80 0x7f56601414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 -- 172.21.15.28:0/3283783229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56601422a0 con 0x7f566006f0a0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f565ea00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f565000c530 tx=0x7f565000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650017070 con 0x7f566006f0a0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650005cf0 con 0x7f566006f0a0 2024-03-21T08:42:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650007270 con 0x7f566006f0a0 2024-03-21T08:42:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5660142580 con 0x7f566006f0a0 2024-03-21T08:42:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5660142b40 con 0x7f566006f0a0 2024-03-21T08:42:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5660073430 con 0x7f566006f0a0 2024-03-21T08:42:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.252+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f565001a070 con 0x7f566006f0a0 2024-03-21T08:42:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.256+0000 7f5656a00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 0x7f563c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.256+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f56500854c0 con 0x7f566006f0a0 2024-03-21T08:42:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.256+0000 7f5657e00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 0x7f563c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.256+0000 7f5657e00640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 0x7f563c069fe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f566013ed90 tx=0x7f5648009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.260+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5650055500 con 0x7f566006f0a0 2024-03-21T08:42:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f566013f2b0 con 0x7f566006f0a0 2024-03-21T08:42:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5656a00640 1 -- 172.21.15.28:0/3283783229 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56500595a0 con 0x7f566006f0a0 2024-03-21T08:42:04.422 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 msgr2=0x7f563c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 0x7f563c069fe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f566013ed90 tx=0x7f5648009040 comp rx=0 tx=0).stop 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 msgr2=0x7f56601fa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f565000c530 tx=0x7f565000c9f0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 shutdown_connections 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f563c067b20 0x7f563c069fe0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f56600cde40 0x7f5660141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5660071a80 0x7f56601414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 --2- 172.21.15.28:0/3283783229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f566006f0a0 0x7f56601fa2b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 >> 172.21.15.28:0/3283783229 conn(0x7f56600674f0 msgr2=0x7f5660144ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 shutdown_connections 2024-03-21T08:42:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.420+0000 7f5665800640 1 -- 172.21.15.28:0/3283783229 wait complete. 2024-03-21T08:42:04.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 Processor -- start 2024-03-21T08:42:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 -- start start 2024-03-21T08:42:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ca00cde40 0x7f8ca0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ca015cbb0 0x7f8ca00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca015c720 con 0x7f8ca00cde40 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca00723a0 con 0x7f8ca015cbb0 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8ca4a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00ce5b0 con 0x7f8ca0071a80 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ca00cde40 0x7f8ca0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c9de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ca00cde40 0x7f8ca0071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53874/0 (socket says 172.21.15.28:53874) 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 -- 172.21.15.28:0/1855106567 learned_addr learned my addr 172.21.15.28:0/1855106567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c9e800640 1 --2- 172.21.15.28:0/1855106567 >> v2:172.21.15.151:3300/0 conn(0x7f8ca015cbb0 0x7f8ca00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 -- 172.21.15.28:0/1855106567 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 msgr2=0x7f8ca006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 -- 172.21.15.28:0/1855106567 >> v2:172.21.15.151:3300/0 conn(0x7f8ca015cbb0 msgr2=0x7f8ca00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1855106567 >> v2:172.21.15.151:3300/0 conn(0x7f8ca015cbb0 0x7f8ca00cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c97e00640 1 -- 172.21.15.28:0/1855106567 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca00ce890 con 0x7f8ca00cde40 2024-03-21T08:42:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.588+0000 7f8c9de00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0071e60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0145950 tx=0x7f8c8801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b931cdec04c8155f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8c9d400640 1 -- 172.21.15.28:0/1855106567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c880ab640 con 0x7f8ca00cde40 2024-03-21T08:42:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8c9d400640 1 -- 172.21.15.28:0/1855106567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c880b0070 con 0x7f8ca00cde40 2024-03-21T08:42:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8c9d400640 1 -- 172.21.15.28:0/1855106567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c880aba00 con 0x7f8ca00cde40 2024-03-21T08:42:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1855106567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 msgr2=0x7f8ca0071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0071e60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0145950 tx=0x7f8c8801c460 comp rx=0 tx=0).stop 2024-03-21T08:42:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1855106567 shutdown_connections 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1855106567 >> v2:172.21.15.151:3300/0 conn(0x7f8ca015cbb0 0x7f8ca00cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0071e60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1855106567 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1855106567 >> 172.21.15.28:0/1855106567 conn(0x7f8ca00674f0 msgr2=0x7f8ca006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1855106567 shutdown_connections 2024-03-21T08:42:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.592+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1855106567 wait complete. 2024-03-21T08:42:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 Processor -- start 2024-03-21T08:42:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- start start 2024-03-21T08:42:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca014e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ca015cbb0 0x7f8ca013be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca0144b20 con 0x7f8ca00cde40 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00cce00 con 0x7f8ca015cbb0 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca006e2b0 con 0x7f8ca0071a80 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53882/0 (socket says 172.21.15.28:53882) 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c9de00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca014e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 -- 172.21.15.28:0/1472491927 learned_addr learned my addr 172.21.15.28:0/1472491927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c9e800640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ca015cbb0 0x7f8ca013be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 -- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 msgr2=0x7f8ca014e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca014e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 -- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ca015cbb0 msgr2=0x7f8ca013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ca015cbb0 0x7f8ca013be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 -- 172.21.15.28:0/1472491927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0141110 con 0x7f8ca00cde40 2024-03-21T08:42:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c97e00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8c8801cd00 tx=0x7f8c8801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c880abdf0 con 0x7f8ca00cde40 2024-03-21T08:42:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c880be3f0 con 0x7f8ca00cde40 2024-03-21T08:42:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c880be7c0 con 0x7f8ca00cde40 2024-03-21T08:42:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca01413d0 con 0x7f8ca00cde40 2024-03-21T08:42:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.596+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca0141990 con 0x7f8ca00cde40 2024-03-21T08:42:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca0073430 con 0x7f8ca00cde40 2024-03-21T08:42:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8c880b43f0 con 0x7f8ca00cde40 2024-03-21T08:42:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8c96000640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 0x7f8c7c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f8c881204d0 con 0x7f8ca00cde40 2024-03-21T08:42:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8c9de00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 0x7f8c7c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.600+0000 7f8c9de00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 0x7f8c7c069fe0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009580 tx=0x7f8c90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.604+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8c880c4060 con 0x7f8ca00cde40 2024-03-21T08:42:04.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.768+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ca004f4f0 con 0x7f8ca00cde40 2024-03-21T08:42:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.768+0000 7f8c96000640 1 -- 172.21.15.28:0/1472491927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8c88016110 con 0x7f8ca00cde40 2024-03-21T08:42:04.772 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 msgr2=0x7f8c7c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 0x7f8c7c069fe0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009580 tx=0x7f8c90009190 comp rx=0 tx=0).stop 2024-03-21T08:42:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 msgr2=0x7f8ca0142190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8c8801cd00 tx=0x7f8c8801aca0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 shutdown_connections 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ca015cbb0 0x7f8ca013be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca00cde40 0x7f8ca0142190 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8c7c067b20 0x7f8c7c069fe0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 --2- 172.21.15.28:0/1472491927 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ca0071a80 0x7f8ca014e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 >> 172.21.15.28:0/1472491927 conn(0x7f8ca00674f0 msgr2=0x7f8ca00b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 shutdown_connections 2024-03-21T08:42:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.772+0000 7f8ca4a00640 1 -- 172.21.15.28:0/1472491927 wait complete. 2024-03-21T08:42:04.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 Processor -- start 2024-03-21T08:42:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 -- start start 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34840b4800 0x7f34840ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3484072190 0x7f34840cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840cd670 con 0x7f34840b4420 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348415bf90 con 0x7f34840b4800 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.932+0000 7f348b000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348415b1d0 con 0x7f3484072190 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34840b4800 0x7f34840ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3483e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f34840b4420 0x7f3484071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55614/0 (socket says 172.21.15.28:55614) 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3483e00640 1 -- 172.21.15.28:0/893024328 learned_addr learned my addr 172.21.15.28:0/893024328 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3489000640 1 --2- 172.21.15.28:0/893024328 >> v2:172.21.15.151:3300/0 conn(0x7f3484072190 0x7f34840cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 -- 172.21.15.28:0/893024328 >> v2:172.21.15.151:3300/0 conn(0x7f3484072190 msgr2=0x7f34840cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 --2- 172.21.15.28:0/893024328 >> v2:172.21.15.151:3300/0 conn(0x7f3484072190 0x7f34840cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 -- 172.21.15.28:0/893024328 >> v1:172.21.15.28:6789/0 conn(0x7f34840b4420 legacy=0x7f3484071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 -- 172.21.15.28:0/893024328 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348415b4b0 con 0x7f34840b4800 2024-03-21T08:42:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f347b400640 1 --2- 172.21.15.28:0/893024328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34840b4800 0x7f34840ccbf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3484155680 tx=0x7f346c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62421b5a3b466fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3483400640 1 -- 172.21.15.28:0/893024328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0ab650 con 0x7f34840b4800 2024-03-21T08:42:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3483400640 1 -- 172.21.15.28:0/893024328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c0b0070 con 0x7f34840b4800 2024-03-21T08:42:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.936+0000 7f3483400640 1 -- 172.21.15.28:0/893024328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0aba10 con 0x7f34840b4800 2024-03-21T08:42:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- 172.21.15.28:0/893024328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34840b4800 msgr2=0x7f34840ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- 172.21.15.28:0/893024328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34840b4800 0x7f34840ccbf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3484155680 tx=0x7f346c01c450 comp rx=0 tx=0).stop 2024-03-21T08:42:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- 172.21.15.28:0/893024328 shutdown_connections 2024-03-21T08:42:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- 172.21.15.28:0/893024328 >> v2:172.21.15.151:3300/0 conn(0x7f3484072190 0x7f34840cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- 172.21.15.28:0/893024328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34840b4800 0x7f34840ccbf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- 172.21.15.28:0/893024328 >> 172.21.15.28:0/893024328 conn(0x7f34840674f0 msgr2=0x7f348406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- 172.21.15.28:0/893024328 shutdown_connections 2024-03-21T08:42:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- 172.21.15.28:0/893024328 wait complete. 2024-03-21T08:42:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 Processor -- start 2024-03-21T08:42:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- start start 2024-03-21T08:42:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34840b4420 0x7f34840c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34840c7e30 0x7f348414c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b3460 con 0x7f3484072190 2024-03-21T08:42:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b32e0 con 0x7f34840c7e30 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f348b000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840b35e0 con 0x7f34840b4420 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53906/0 (socket says 172.21.15.28:53906) 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 -- 172.21.15.28:0/3334636458 learned_addr learned my addr 172.21.15.28:0/3334636458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f347b400640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34840b4420 0x7f34840c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3489000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34840c7e30 0x7f348414c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 -- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34840b4420 msgr2=0x7f34840c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34840b4420 0x7f34840c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 -- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34840c7e30 msgr2=0x7f348414c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:04.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34840c7e30 0x7f348414c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.940+0000 7f3483e00640 1 -- 172.21.15.28:0/3334636458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348414b860 con 0x7f3484072190 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3483e00640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f347400cb10 tx=0x7f347400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474017070 con 0x7f3484072190 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400ac70 con 0x7f3484072190 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474012620 con 0x7f3484072190 2024-03-21T08:42:04.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34840c82b0 con 0x7f3484072190 2024-03-21T08:42:04.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348414d1f0 con 0x7f3484072190 2024-03-21T08:42:04.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484073430 con 0x7f3484072190 2024-03-21T08:42:04.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f347401a070 con 0x7f3484072190 2024-03-21T08:42:04.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 0x7f3460069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:04.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f3474085fe0 con 0x7f3484072190 2024-03-21T08:42:04.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.944+0000 7f347b400640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 0x7f3460069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:04.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.948+0000 7f347b400640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 0x7f3460069fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f346c0af000 tx=0x7f346c0b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:04.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:04.948+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f348414b860 con 0x7f3484072190 2024-03-21T08:42:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f348404f4f0 con 0x7f3484072190 2024-03-21T08:42:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f3480c00640 1 -- 172.21.15.28:0/3334636458 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3474055fa0 con 0x7f3484072190 2024-03-21T08:42:05.113 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 msgr2=0x7f3460069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 0x7f3460069fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f346c0af000 tx=0x7f346c0b6040 comp rx=0 tx=0).stop 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 msgr2=0x7f348414b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f347400cb10 tx=0x7f347400a060 comp rx=0 tx=0).stop 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 shutdown_connections 2024-03-21T08:42:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f34840c7e30 0x7f348414c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3460067b20 0x7f3460069fe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f34840b4420 0x7f34840c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 --2- 172.21.15.28:0/3334636458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3484072190 0x7f348414b320 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 >> 172.21.15.28:0/3334636458 conn(0x7f34840674f0 msgr2=0x7f348415e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 shutdown_connections 2024-03-21T08:42:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.112+0000 7f348b000640 1 -- 172.21.15.28:0/3334636458 wait complete. 2024-03-21T08:42:05.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 Processor -- start 2024-03-21T08:42:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 -- start start 2024-03-21T08:42:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e4415e660 0x7f4e4406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4406c280 0x7f4e4415ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4406e240 con 0x7f4e4406c700 2024-03-21T08:42:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4406cc60 con 0x7f4e4415e660 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e43200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4406cf40 con 0x7f4e4406c280 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e4415e660 0x7f4e4406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e42800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4e4406c700 0x7f4e4415f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55630/0 (socket says 172.21.15.28:55630) 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e42800640 1 -- 172.21.15.28:0/1609277682 learned_addr learned my addr 172.21.15.28:0/1609277682 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e4415e660 0x7f4e4406dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53920/0 (socket says 172.21.15.28:53920) 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e39400640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4406c280 0x7f4e4415ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 -- 172.21.15.28:0/1609277682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4406c280 msgr2=0x7f4e4415ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4406c280 0x7f4e4415ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 -- 172.21.15.28:0/1609277682 >> v1:172.21.15.28:6789/0 conn(0x7f4e4406c700 legacy=0x7f4e4415f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41400640 1 -- 172.21.15.28:0/1609277682 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716299476 0 0) 0x7f4e4406e240 con 0x7f4e4406c700 2024-03-21T08:42:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.276+0000 7f4e41e00640 1 -- 172.21.15.28:0/1609277682 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4406d220 con 0x7f4e4415e660 2024-03-21T08:42:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e41e00640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4415e660 0x7f4e4406dd00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009dc0 tx=0x7f4e3401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6339b61b3a9538 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e41400640 1 -- 172.21.15.28:0/1609277682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340b2070 con 0x7f4e4415e660 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e41400640 1 -- 172.21.15.28:0/1609277682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e340a5870 con 0x7f4e4415e660 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e41400640 1 -- 172.21.15.28:0/1609277682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340a5c40 con 0x7f4e4415e660 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 -- 172.21.15.28:0/1609277682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4415e660 msgr2=0x7f4e4406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4415e660 0x7f4e4406dd00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009dc0 tx=0x7f4e3401c8c0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 -- 172.21.15.28:0/1609277682 shutdown_connections 2024-03-21T08:42:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4406c280 0x7f4e4415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 --2- 172.21.15.28:0/1609277682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4415e660 0x7f4e4406dd00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 -- 172.21.15.28:0/1609277682 >> 172.21.15.28:0/1609277682 conn(0x7f4e440b3440 msgr2=0x7f4e440b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 -- 172.21.15.28:0/1609277682 shutdown_connections 2024-03-21T08:42:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.280+0000 7f4e43200640 1 -- 172.21.15.28:0/1609277682 wait complete. 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 Processor -- start 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- start start 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 0x7f4e441fa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 0x7f4e441fac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440cf7a0 con 0x7f4e4406c280 2024-03-21T08:42:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4415f390 con 0x7f4e4406c700 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440cf920 con 0x7f4e4415e660 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53932/0 (socket says 172.21.15.28:53932) 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 -- 172.21.15.28:0/1927743638 learned_addr learned my addr 172.21.15.28:0/1927743638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e39400640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 0x7f4e441fa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e42800640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 0x7f4e441fac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 -- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 msgr2=0x7f4e441fac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 0x7f4e441fac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 -- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 msgr2=0x7f4e441fa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 0x7f4e441fa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 -- 172.21.15.28:0/1927743638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440c5470 con 0x7f4e4406c280 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e42800640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 0x7f4e441fac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e39400640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 0x7f4e441fa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e41e00640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009dc0 tx=0x7f4e34019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340bd070 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34003ba0 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340b2040 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e440c5750 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.284+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e440b6de0 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4404f410 con 0x7f4e4406c280 2024-03-21T08:42:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4e340a56f0 con 0x7f4e4406c280 2024-03-21T08:42:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 0x7f4e20069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f4e34121250 con 0x7f4e4406c280 2024-03-21T08:42:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e39400640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 0x7f4e20069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.288+0000 7f4e39400640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 0x7f4e20069fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4e440cb9f0 tx=0x7f4e2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.292+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4e34016540 con 0x7f4e4406c280 2024-03-21T08:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.468+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4e440c5920 con 0x7f4e4406c280 2024-03-21T08:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.468+0000 7f4e3aa00640 1 -- 172.21.15.28:0/1927743638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4e340f1840 con 0x7f4e4406c280 2024-03-21T08:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 msgr2=0x7f4e20069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 0x7f4e20069fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4e440cb9f0 tx=0x7f4e2c008040 comp rx=0 tx=0).stop 2024-03-21T08:42:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 msgr2=0x7f4e440b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009dc0 tx=0x7f4e34019660 comp rx=0 tx=0).stop 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 shutdown_connections 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4e20067b20 0x7f4e20069fe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4e4415e660 0x7f4e441fac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4e4406c700 0x7f4e441fa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 --2- 172.21.15.28:0/1927743638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e4406c280 0x7f4e440b68a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 >> 172.21.15.28:0/1927743638 conn(0x7f4e440b3440 msgr2=0x7f4e440b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 shutdown_connections 2024-03-21T08:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.472+0000 7f4e43200640 1 -- 172.21.15.28:0/1927743638 wait complete. 2024-03-21T08:42:05.492 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi151.front.sepia.ceph.com... 2024-03-21T08:42:05.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:05.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-21T08:42:05.520 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:05.520 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:42:05.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:05.520 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-21T08:42:05.584 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:05.584 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:05.593 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:05.593 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:05.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:05.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 Processor -- start 2024-03-21T08:42:05.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 -- start start 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb1e015e0a0 0x7fb1e00ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00b5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e00ceee0 con 0x7fb1e015dcc0 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e00cf1c0 con 0x7fb1e015e0a0 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e6e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e006c760 con 0x7fb1e00ce9a0 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb1e015e0a0 0x7fb1e00ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53934/0 (socket says 172.21.15.28:53934) 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1dfe00640 1 -- 172.21.15.28:0/3975459266 learned_addr learned my addr 172.21.15.28:0/3975459266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1e4e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 -- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 msgr2=0x7fb1e00b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 -- 172.21.15.28:0/3975459266 >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 msgr2=0x7fb1e006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/3975459266 >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1d7e00640 1 -- 172.21.15.28:0/3975459266 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e006ca40 con 0x7fb1e015e0a0 2024-03-21T08:42:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.828+0000 7fb1dfe00640 1 --2- 172.21.15.28:0/3975459266 >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00ce460 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00b6510 tx=0x7fb1c801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6080c024478ba1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1df400640 1 -- 172.21.15.28:0/3975459266 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c80b0070 con 0x7fb1e015e0a0 2024-03-21T08:42:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1df400640 1 -- 172.21.15.28:0/3975459266 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c80ab600 con 0x7fb1e015e0a0 2024-03-21T08:42:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1df400640 1 -- 172.21.15.28:0/3975459266 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c80ab9d0 con 0x7fb1e015e0a0 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 -- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 msgr2=0x7fb1e00ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00ce460 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00b6510 tx=0x7fb1c801c3c0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 -- 172.21.15.28:0/3975459266 shutdown_connections 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00b5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/3975459266 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00ce460 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/3975459266 >> v2:172.21.15.28:3300/0 conn(0x7fb1e015dcc0 0x7fb1e006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 -- 172.21.15.28:0/3975459266 >> 172.21.15.28:0/3975459266 conn(0x7fb1e0072fe0 msgr2=0x7fb1e00d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 -- 172.21.15.28:0/3975459266 shutdown_connections 2024-03-21T08:42:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.832+0000 7fb1e6e00640 1 -- 172.21.15.28:0/3975459266 wait complete. 2024-03-21T08:42:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 Processor -- start 2024-03-21T08:42:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- start start 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00d2ad0 con 0x7fb1e015dcc0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00d2950 con 0x7fb1e015e0a0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0161a70 con 0x7fb1e00ce9a0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53940/0 (socket says 172.21.15.28:53940) 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 -- 172.21.15.28:0/2775288334 learned_addr learned my addr 172.21.15.28:0/2775288334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e4e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1dfe00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 -- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 msgr2=0x7fb1e00cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 -- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 msgr2=0x7fb1e00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 -- 172.21.15.28:0/2775288334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e01588b0 con 0x7fb1e015dcc0 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1dfe00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:05.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1d7e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb1c801aa70 tx=0x7fb1c8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c80ba020 con 0x7fb1e015dcc0 2024-03-21T08:42:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c80b5460 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c80b0040 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e0158b90 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e0159150 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.836+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e004f410 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.840+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb1c80bd020 con 0x7fb1e015dcc0 2024-03-21T08:42:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.840+0000 7fb1dcc00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 0x7fb1bc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.840+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fb1c8120650 con 0x7fb1e015dcc0 2024-03-21T08:42:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.840+0000 7fb1dfe00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 0x7fb1bc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.840+0000 7fb1dfe00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 0x7fb1bc069fe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009630 tx=0x7fb1d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:05.844+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb1c80c4030 con 0x7fb1e015dcc0 2024-03-21T08:42:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e00d17a0 con 0x7fb1e015dcc0 2024-03-21T08:42:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1dcc00640 1 -- 172.21.15.28:0/2775288334 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7fb1c8016070 con 0x7fb1e015dcc0 2024-03-21T08:42:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:06.010 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 msgr2=0x7fb1bc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 0x7fb1bc069fe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009630 tx=0x7fb1d0008040 comp rx=0 tx=0).stop 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 msgr2=0x7fb1e00cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb1c801aa70 tx=0x7fb1c8019160 comp rx=0 tx=0).stop 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 shutdown_connections 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1e015e0a0 0x7fb1e00cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1e015dcc0 0x7fb1e00cd370 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb1bc067b20 0x7fb1bc069fe0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 --2- 172.21.15.28:0/2775288334 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1e00ce9a0 0x7fb1e00cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 >> 172.21.15.28:0/2775288334 conn(0x7fb1e0072fe0 msgr2=0x7fb1e00b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.008+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 shutdown_connections 2024-03-21T08:42:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.012+0000 7fb1e6e00640 1 -- 172.21.15.28:0/2775288334 wait complete. 2024-03-21T08:42:06.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:06.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 Processor -- start 2024-03-21T08:42:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 -- start start 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1aac06f0a0 0x7f1aac0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aac06f500 con 0x7f1aac15cbb0 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aac0ce5b0 con 0x7f1aac071a80 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aac0ce380 con 0x7f1aac06f0a0 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aab400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1aac06f0a0 0x7f1aac0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aabe00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1aac071a80 0x7f1aac071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:48966/0 (socket says 172.21.15.28:48966) 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aabe00640 1 -- 172.21.15.28:0/2089577745 learned_addr learned my addr 172.21.15.28:0/2089577745 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1ab1200640 1 --1- 172.21.15.28:0/2089577745 >> v1:172.21.15.28:6789/0 conn(0x7f1aac15cbb0 0x7f1aac0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55638/0 (socket says 172.21.15.28:55638) 2024-03-21T08:42:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aab400640 1 -- 172.21.15.28:0/2089577745 >> v1:172.21.15.151:6789/0 conn(0x7f1aac071a80 legacy=0x7f1aac071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aab400640 1 -- 172.21.15.28:0/2089577745 >> v1:172.21.15.28:6789/0 conn(0x7f1aac15cbb0 legacy=0x7f1aac0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aab400640 1 -- 172.21.15.28:0/2089577745 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac0ce890 con 0x7f1aac06f0a0 2024-03-21T08:42:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.176+0000 7f1aaaa00640 1 -- 172.21.15.28:0/2089577745 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681525891 0 0) 0x7f1aac0ce5b0 con 0x7f1aac071a80 2024-03-21T08:42:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1aab400640 1 --2- 172.21.15.28:0/2089577745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac06f0a0 0x7f1aac0cde40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1a980097c0 tx=0x7f1a9801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac77e04d32a6d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1aaaa00640 1 -- 172.21.15.28:0/2089577745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980b0070 con 0x7f1aac06f0a0 2024-03-21T08:42:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1aaaa00640 1 -- 172.21.15.28:0/2089577745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a980ab600 con 0x7f1aac06f0a0 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1aaaa00640 1 -- 172.21.15.28:0/2089577745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980ab9d0 con 0x7f1aac06f0a0 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- 172.21.15.28:0/2089577745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac06f0a0 msgr2=0x7f1aac0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 --2- 172.21.15.28:0/2089577745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac06f0a0 0x7f1aac0cde40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1a980097c0 tx=0x7f1a9801c410 comp rx=0 tx=0).stop 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- 172.21.15.28:0/2089577745 shutdown_connections 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 --2- 172.21.15.28:0/2089577745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac06f0a0 0x7f1aac0cde40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- 172.21.15.28:0/2089577745 >> 172.21.15.28:0/2089577745 conn(0x7f1aac0674f0 msgr2=0x7f1aac06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- 172.21.15.28:0/2089577745 shutdown_connections 2024-03-21T08:42:06.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- 172.21.15.28:0/2089577745 wait complete. 2024-03-21T08:42:06.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 Processor -- start 2024-03-21T08:42:06.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.180+0000 7f1ab3200640 1 -- start start 2024-03-21T08:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aac06f0a0 0x7f1aac1482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aac071a80 0x7f1aac138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac16a0b0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac169f30 con 0x7f1aac071a80 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac16a230 con 0x7f1aac06f0a0 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1aabe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aac06f0a0 0x7f1aac1482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53956/0 (socket says 172.21.15.28:53956) 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 -- 172.21.15.28:0/4237111363 learned_addr learned my addr 172.21.15.28:0/4237111363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1aab400640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aac071a80 0x7f1aac138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 -- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aac06f0a0 msgr2=0x7f1aac1482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aac06f0a0 0x7f1aac1482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 -- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aac071a80 msgr2=0x7f1aac138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aac071a80 0x7f1aac138780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 -- 172.21.15.28:0/4237111363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac141840 con 0x7f1aac15cbb0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab1200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c760 tx=0x7f1aa400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4017070 con 0x7f1aac15cbb0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa400a640 con 0x7f1aac15cbb0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa400a9f0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac141ae0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac1420a0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.184+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac073430 con 0x7f1aac15cbb0 2024-03-21T08:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.188+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1aa401a070 con 0x7f1aac15cbb0 2024-03-21T08:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.188+0000 7f1aa3e00640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 0x7f1a8c06a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.188+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f1aa4085820 con 0x7f1aac15cbb0 2024-03-21T08:42:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.188+0000 7f1aabe00640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 0x7f1a8c06a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.188+0000 7f1aabe00640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 0x7f1a8c06a030 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c000f70 tx=0x7f1a9c007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.192+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1aa40557b0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1aac04f4f0 con 0x7f1aac15cbb0 2024-03-21T08:42:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1aa3e00640 1 -- 172.21.15.28:0/4237111363 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1aa4059850 con 0x7f1aac15cbb0 2024-03-21T08:42:06.357 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 msgr2=0x7f1a8c06a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 0x7f1a8c06a030 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c000f70 tx=0x7f1a9c007240 comp rx=0 tx=0).stop 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 msgr2=0x7f1aac142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c760 tx=0x7f1aa400cc20 comp rx=0 tx=0).stop 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 shutdown_connections 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aac15cbb0 0x7f1aac142b80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aac071a80 0x7f1aac138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b70 0x7f1a8c06a030 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 --2- 172.21.15.28:0/4237111363 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aac06f0a0 0x7f1aac1482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 >> 172.21.15.28:0/4237111363 conn(0x7f1aac0674f0 msgr2=0x7f1aac0b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 shutdown_connections 2024-03-21T08:42:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.356+0000 7f1ab3200640 1 -- 172.21.15.28:0/4237111363 wait complete. 2024-03-21T08:42:06.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 Processor -- start 2024-03-21T08:42:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 -- start start 2024-03-21T08:42:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ec06f500 con 0x7f75ec15cbb0 2024-03-21T08:42:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ec0ce5b0 con 0x7f75ec071a80 2024-03-21T08:42:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75f1e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec0ce380 con 0x7f75ec06f0a0 2024-03-21T08:42:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75eb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f75ec15cbb0 0x7f75ec0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55644/0 (socket says 172.21.15.28:55644) 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75eb400640 1 -- 172.21.15.28:0/2511178629 learned_addr learned my addr 172.21.15.28:0/2511178629 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75e2000640 1 --2- 172.21.15.28:0/2511178629 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654283867 0 0) 0x7f75ec0ce5b0 con 0x7f75ec071a80 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d0003530 con 0x7f75ec071a80 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800997391 0 0) 0x7f75ec06f500 con 0x7f75ec15cbb0 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ec0ce5b0 con 0x7f75ec15cbb0 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75e2000640 1 -- 172.21.15.28:0/2511178629 >> v1:172.21.15.151:6789/0 conn(0x7f75ec071a80 legacy=0x7f75ec071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75e2000640 1 -- 172.21.15.28:0/2511178629 >> v1:172.21.15.28:6789/0 conn(0x7f75ec15cbb0 legacy=0x7f75ec0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.520+0000 7f75e2000640 1 -- 172.21.15.28:0/2511178629 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec0ce890 con 0x7f75ec06f0a0 2024-03-21T08:42:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75e2000640 1 --2- 172.21.15.28:0/2511178629 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec0cde40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da231669029fa598 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40b2070 con 0x7f75ec06f0a0 2024-03-21T08:42:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d40a9070 con 0x7f75ec06f0a0 2024-03-21T08:42:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75ea000640 1 -- 172.21.15.28:0/2511178629 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40ad880 con 0x7f75ec06f0a0 2024-03-21T08:42:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 -- 172.21.15.28:0/2511178629 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 msgr2=0x7f75ec0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 --2- 172.21.15.28:0/2511178629 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec0cde40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d401c450 comp rx=0 tx=0).stop 2024-03-21T08:42:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 -- 172.21.15.28:0/2511178629 shutdown_connections 2024-03-21T08:42:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 --2- 172.21.15.28:0/2511178629 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec0cde40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 -- 172.21.15.28:0/2511178629 >> 172.21.15.28:0/2511178629 conn(0x7f75ec0674f0 msgr2=0x7f75ec06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 -- 172.21.15.28:0/2511178629 shutdown_connections 2024-03-21T08:42:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.524+0000 7f75f1e00640 1 -- 172.21.15.28:0/2511178629 wait complete. 2024-03-21T08:42:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 Processor -- start 2024-03-21T08:42:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- start start 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75ec15cbb0 0x7f75ec137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec16a0b0 con 0x7f75ec071a80 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec169f30 con 0x7f75ec15cbb0 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec16a230 con 0x7f75ec06f0a0 2024-03-21T08:42:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53964/0 (socket says 172.21.15.28:53964) 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 -- 172.21.15.28:0/3266741126 learned_addr learned my addr 172.21.15.28:0/3266741126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75eaa00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75eb400640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75ec15cbb0 0x7f75ec137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 -- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 msgr2=0x7f75ec134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 -- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75ec15cbb0 msgr2=0x7f75ec137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75ec15cbb0 0x7f75ec137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 -- 172.21.15.28:0/3266741126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec137270 con 0x7f75ec071a80 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e2000640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40ba020 con 0x7f75ec071a80 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d40add10 con 0x7f75ec071a80 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ec137550 con 0x7f75ec071a80 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40b2040 con 0x7f75ec071a80 2024-03-21T08:42:06.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.528+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec1596c0 con 0x7f75ec071a80 2024-03-21T08:42:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ec073430 con 0x7f75ec071a80 2024-03-21T08:42:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f75d40bd020 con 0x7f75ec071a80 2024-03-21T08:42:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75e3e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 0x7f75c8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f75d411fce0 con 0x7f75ec071a80 2024-03-21T08:42:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75eaa00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 0x7f75c8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.532+0000 7f75eaa00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 0x7f75c8069fe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f75dc001570 tx=0x7f75dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.536+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f75d40c4030 con 0x7f75ec071a80 2024-03-21T08:42:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75ec0af4b0 con 0x7f75ec071a80 2024-03-21T08:42:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75e3e00640 1 -- 172.21.15.28:0/3266741126 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f75d40f4e20 con 0x7f75ec071a80 2024-03-21T08:42:06.701 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 msgr2=0x7f75c8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 0x7f75c8069fe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f75dc001570 tx=0x7f75dc009040 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 msgr2=0x7f75ec138ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d40191f0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 shutdown_connections 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f75ec15cbb0 0x7f75ec137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ec071a80 0x7f75ec138ec0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75c8067b20 0x7f75c8069fe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 --2- 172.21.15.28:0/3266741126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f75ec06f0a0 0x7f75ec134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 >> 172.21.15.28:0/3266741126 conn(0x7f75ec0674f0 msgr2=0x7f75ec0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 shutdown_connections 2024-03-21T08:42:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.700+0000 7f75f1e00640 1 -- 172.21.15.28:0/3266741126 wait complete. 2024-03-21T08:42:06.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:06.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 Processor -- start 2024-03-21T08:42:06.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 -- start start 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f07500b4420 0x7f0750071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f07500cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075015bf90 con 0x7f07500b4800 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075015b1d0 con 0x7f07500b4420 2024-03-21T08:42:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0758600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075015b4b0 con 0x7f07500ccbf0 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0755200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f07500cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0755200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53968/0 (socket says 172.21.15.28:53968) 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0755200640 1 -- 172.21.15.28:0/1925432113 learned_addr learned my addr 172.21.15.28:0/1925432113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0755c00640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.151:3300/0 conn(0x7f07500b4420 0x7f0750071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 -- 172.21.15.28:0/1925432113 >> v2:172.21.15.151:3300/0 conn(0x7f07500b4420 msgr2=0x7f0750071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.151:3300/0 conn(0x7f07500b4420 0x7f0750071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 -- 172.21.15.28:0/1925432113 >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 msgr2=0x7f0750071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0756600640 1 -- 172.21.15.28:0/1925432113 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075015b790 con 0x7f07500ccbf0 2024-03-21T08:42:06.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.864+0000 7f0755200640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:06.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0756600640 1 --2- 172.21.15.28:0/1925432113 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f07500cd070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f075006ecd0 tx=0x7f074801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aebeb19097ff05b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f074fe00640 1 -- 172.21.15.28:0/1925432113 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07480b2070 con 0x7f07500ccbf0 2024-03-21T08:42:06.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f074fe00640 1 -- 172.21.15.28:0/1925432113 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07480a5920 con 0x7f07500ccbf0 2024-03-21T08:42:06.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f074fe00640 1 -- 172.21.15.28:0/1925432113 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07480a5cf0 con 0x7f07500ccbf0 2024-03-21T08:42:06.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 -- 172.21.15.28:0/1925432113 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 msgr2=0x7f07500cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 --2- 172.21.15.28:0/1925432113 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f07500cd070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f075006ecd0 tx=0x7f074801c900 comp rx=0 tx=0).stop 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 -- 172.21.15.28:0/1925432113 shutdown_connections 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 --2- 172.21.15.28:0/1925432113 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f07500cd070 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.28:3300/0 conn(0x7f07500b4800 0x7f0750071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 --2- 172.21.15.28:0/1925432113 >> v2:172.21.15.151:3300/0 conn(0x7f07500b4420 0x7f0750071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 -- 172.21.15.28:0/1925432113 >> 172.21.15.28:0/1925432113 conn(0x7f07500674f0 msgr2=0x7f075006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 -- 172.21.15.28:0/1925432113 shutdown_connections 2024-03-21T08:42:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.868+0000 7f0758600640 1 -- 172.21.15.28:0/1925432113 wait complete. 2024-03-21T08:42:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 Processor -- start 2024-03-21T08:42:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- start start 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07500b4420 0x7f0750158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f0750156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075015efb0 con 0x7f075015c610 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075015ee30 con 0x7f07500b4420 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075015f130 con 0x7f07500ccbf0 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53970/0 (socket says 172.21.15.28:53970) 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 -- 172.21.15.28:0/3380991165 learned_addr learned my addr 172.21.15.28:0/3380991165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0755c00640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07500b4420 0x7f0750158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0755200640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f0750156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 -- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 msgr2=0x7f0750156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f0750156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 -- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07500b4420 msgr2=0x7f0750158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07500b4420 0x7f0750158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 -- 172.21.15.28:0/3380991165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07501569c0 con 0x7f075015c610 2024-03-21T08:42:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0756600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f074801afb0 tx=0x7f07480196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07480b2050 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07480be070 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750156ca0 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07480ad890 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.872+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07501f4b20 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0750073430 con 0x7f075015c610 2024-03-21T08:42:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f07480ada30 con 0x7f075015c610 2024-03-21T08:42:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f074d600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 0x7f073006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f07481214d0 con 0x7f075015c610 2024-03-21T08:42:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f0755c00640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 0x7f073006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:06.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.876+0000 7f0755c00640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 0x7f073006a010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0744009b60 tx=0x7f0744009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:06.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:06.880+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0748016520 con 0x7f075015c610 2024-03-21T08:42:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.040+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f075004f4f0 con 0x7f075015c610 2024-03-21T08:42:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.040+0000 7f074d600640 1 -- 172.21.15.28:0/3380991165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f07480f1470 con 0x7f075015c610 2024-03-21T08:42:07.044 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 msgr2=0x7f073006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 0x7f073006a010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0744009b60 tx=0x7f0744009190 comp rx=0 tx=0).stop 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 msgr2=0x7f075013bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f074801afb0 tx=0x7f07480196a0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 shutdown_connections 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075015c610 0x7f075013bdc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0730067b50 0x7f073006a010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07500ccbf0 0x7f0750156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 --2- 172.21.15.28:0/3380991165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07500b4420 0x7f0750158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 >> 172.21.15.28:0/3380991165 conn(0x7f07500674f0 msgr2=0x7f0750154540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 shutdown_connections 2024-03-21T08:42:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.044+0000 7f0758600640 1 -- 172.21.15.28:0/3380991165 wait complete. 2024-03-21T08:42:07.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:07.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 Processor -- start 2024-03-21T08:42:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 -- start start 2024-03-21T08:42:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f07f4071a80 0x7f07f406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07f415cbb0 0x7f07f40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40ce5b0 con 0x7f07f40cde40 2024-03-21T08:42:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f415c720 con 0x7f07f415cbb0 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f8a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f40ce890 con 0x7f07f4071a80 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07f415cbb0 0x7f07f40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07f415cbb0 0x7f07f40cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53978/0 (socket says 172.21.15.28:53978) 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 -- 172.21.15.28:0/265301556 learned_addr learned my addr 172.21.15.28:0/265301556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07e9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f07f40cde40 0x7f07f4071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55646/0 (socket says 172.21.15.28:55646) 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f1e00640 1 --2- 172.21.15.28:0/265301556 >> v2:172.21.15.151:3300/0 conn(0x7f07f4071a80 0x7f07f406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 -- 172.21.15.28:0/265301556 >> v2:172.21.15.151:3300/0 conn(0x7f07f4071a80 msgr2=0x7f07f406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 --2- 172.21.15.28:0/265301556 >> v2:172.21.15.151:3300/0 conn(0x7f07f4071a80 0x7f07f406f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 -- 172.21.15.28:0/265301556 >> v1:172.21.15.28:6789/0 conn(0x7f07f40cde40 legacy=0x7f07f4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.208+0000 7f07f2800640 1 -- 172.21.15.28:0/265301556 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f415c3b0 con 0x7f07f415cbb0 2024-03-21T08:42:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f2800640 1 --2- 172.21.15.28:0/265301556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f415cbb0 0x7f07f40cc4b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f07f40b4f30 tx=0x7f07ec01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a123fac57dbc4012 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f1400640 1 -- 172.21.15.28:0/265301556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0aa610 con 0x7f07f415cbb0 2024-03-21T08:42:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f1400640 1 -- 172.21.15.28:0/265301556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec0af070 con 0x7f07f415cbb0 2024-03-21T08:42:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f1400640 1 -- 172.21.15.28:0/265301556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0aa9d0 con 0x7f07f415cbb0 2024-03-21T08:42:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- 172.21.15.28:0/265301556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f415cbb0 msgr2=0x7f07f40cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 --2- 172.21.15.28:0/265301556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f415cbb0 0x7f07f40cc4b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f07f40b4f30 tx=0x7f07ec01c490 comp rx=0 tx=0).stop 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- 172.21.15.28:0/265301556 shutdown_connections 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 --2- 172.21.15.28:0/265301556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f415cbb0 0x7f07f40cc4b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 --2- 172.21.15.28:0/265301556 >> v2:172.21.15.151:3300/0 conn(0x7f07f4071a80 0x7f07f406f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- 172.21.15.28:0/265301556 >> 172.21.15.28:0/265301556 conn(0x7f07f40674f0 msgr2=0x7f07f406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- 172.21.15.28:0/265301556 shutdown_connections 2024-03-21T08:42:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- 172.21.15.28:0/265301556 wait complete. 2024-03-21T08:42:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 Processor -- start 2024-03-21T08:42:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.212+0000 7f07f8a00640 1 -- start start 2024-03-21T08:42:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 0x7f07f40b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07f415cbb0 0x7f07f40b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f4168f20 con 0x7f07f4071a80 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f40ce220 con 0x7f07f40cde40 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f41690a0 con 0x7f07f415cbb0 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53986/0 (socket says 172.21.15.28:53986) 2024-03-21T08:42:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 -- 172.21.15.28:0/4176036067 learned_addr learned my addr 172.21.15.28:0/4176036067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f2800640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07f415cbb0 0x7f07f40b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07e9400640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 0x7f07f40b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 -- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07f415cbb0 msgr2=0x7f07f40b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07f415cbb0 0x7f07f40b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 -- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 msgr2=0x7f07f40b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 0x7f07f40b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 -- 172.21.15.28:0/4176036067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40b1780 con 0x7f07f4071a80 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07e9400640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 0x7f07f40b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f1e00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f07e400efc0 tx=0x7f07e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e4018020 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e4014480 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e4014850 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f40b1a60 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f40b2020 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f404f4f0 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.216+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f07e401b020 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.220+0000 7f07eaa00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 0x7f07d0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.220+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f07e4086e70 con 0x7f07f4071a80 2024-03-21T08:42:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.220+0000 7f07e9400640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 0x7f07d0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.220+0000 7f07e9400640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 0x7f07d0069fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07f40cb460 tx=0x7f07dc007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.224+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07e4022070 con 0x7f07f4071a80 2024-03-21T08:42:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f07f4073410 con 0x7f07f4071a80 2024-03-21T08:42:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07eaa00640 1 -- 172.21.15.28:0/4176036067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f07e4010040 con 0x7f07f4071a80 2024-03-21T08:42:07.393 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 msgr2=0x7f07d0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 0x7f07d0069fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07f40cb460 tx=0x7f07dc007d70 comp rx=0 tx=0).stop 2024-03-21T08:42:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 msgr2=0x7f07f413d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f07e400efc0 tx=0x7f07e400c490 comp rx=0 tx=0).stop 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 shutdown_connections 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f07d0067b20 0x7f07d0069fe0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f07f415cbb0 0x7f07f40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f07f40cde40 0x7f07f40b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 --2- 172.21.15.28:0/4176036067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07f4071a80 0x7f07f413d760 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 >> 172.21.15.28:0/4176036067 conn(0x7f07f40674f0 msgr2=0x7f07f40ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 shutdown_connections 2024-03-21T08:42:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.392+0000 7f07f8a00640 1 -- 172.21.15.28:0/4176036067 wait complete. 2024-03-21T08:42:07.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:07.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.552+0000 7f1e33200640 1 Processor -- start 2024-03-21T08:42:07.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.552+0000 7f1e33200640 1 -- start start 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e33200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 0x7f1e2c06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e33200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0ce5b0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e33200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0ce890 con 0x7f1e2c15cbb0 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e33200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c15c720 con 0x7f1e2c071a80 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e23400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1e2c0cde40 0x7f1e2c071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55658/0 (socket says 172.21.15.28:55658) 2024-03-21T08:42:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e23400640 1 -- 172.21.15.28:0/2897399382 learned_addr learned my addr 172.21.15.28:0/2897399382 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2be00640 1 --2- 172.21.15.28:0/2897399382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 0x7f1e2c06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751324074 0 0) 0x7f1e2c0ce890 con 0x7f1e2c15cbb0 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e10003530 con 0x7f1e2c15cbb0 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877991917 0 0) 0x7f1e2c0ce5b0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e2c0ce890 con 0x7f1e2c0cde40 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2be00640 1 -- 172.21.15.28:0/2897399382 >> v1:172.21.15.151:6789/0 conn(0x7f1e2c15cbb0 legacy=0x7f1e2c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2be00640 1 -- 172.21.15.28:0/2897399382 >> v1:172.21.15.28:6789/0 conn(0x7f1e2c0cde40 legacy=0x7f1e2c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.556+0000 7f1e2be00640 1 -- 172.21.15.28:0/2897399382 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c15c3b0 con 0x7f1e2c071a80 2024-03-21T08:42:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e2be00640 1 --2- 172.21.15.28:0/2897399382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 0x7f1e2c06f0a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009dc0 tx=0x7f1e1c01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39d5c15d19e53bb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0b2070 con 0x7f1e2c071a80 2024-03-21T08:42:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c0a59e0 con 0x7f1e2c071a80 2024-03-21T08:42:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e2b400640 1 -- 172.21.15.28:0/2897399382 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0a5d90 con 0x7f1e2c071a80 2024-03-21T08:42:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- 172.21.15.28:0/2897399382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 msgr2=0x7f1e2c06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 --2- 172.21.15.28:0/2897399382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 0x7f1e2c06f0a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009dc0 tx=0x7f1e1c01c900 comp rx=0 tx=0).stop 2024-03-21T08:42:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- 172.21.15.28:0/2897399382 shutdown_connections 2024-03-21T08:42:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 --2- 172.21.15.28:0/2897399382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c071a80 0x7f1e2c06f0a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- 172.21.15.28:0/2897399382 >> 172.21.15.28:0/2897399382 conn(0x7f1e2c0674f0 msgr2=0x7f1e2c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- 172.21.15.28:0/2897399382 shutdown_connections 2024-03-21T08:42:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- 172.21.15.28:0/2897399382 wait complete. 2024-03-21T08:42:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 Processor -- start 2024-03-21T08:42:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.560+0000 7f1e33200640 1 -- start start 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e2c071a80 0x7f1e2c1364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 0x7f1e2c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c16a0b0 con 0x7f1e2c15cbb0 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c169f30 con 0x7f1e2c071a80 2024-03-21T08:42:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c16a230 con 0x7f1e2c0cde40 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e31200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 0x7f1e2c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e2be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e2c071a80 0x7f1e2c1364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e31200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54002/0 (socket says 172.21.15.28:54002) 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e31200640 1 -- 172.21.15.28:0/3928960219 learned_addr learned my addr 172.21.15.28:0/3928960219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 -- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e2c071a80 msgr2=0x7f1e2c1364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e2c071a80 0x7f1e2c1364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 -- 172.21.15.28:0/3928960219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 msgr2=0x7f1e2c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 -- 172.21.15.28:0/3928960219 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c132e30 con 0x7f1e2c0cde40 2024-03-21T08:42:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e31200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e23400640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 0x7f1e2c15ad50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400c3c0 tx=0x7f1e1400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14017070 con 0x7f1e2c0cde40 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e14005cf0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c1330d0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140072d0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c133690 con 0x7f1e2c0cde40 2024-03-21T08:42:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c073430 con 0x7f1e2c0cde40 2024-03-21T08:42:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.564+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1e1401a070 con 0x7f1e2c0cde40 2024-03-21T08:42:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.568+0000 7f1e28c00640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 0x7f1e08069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.568+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f1e140856c0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.568+0000 7f1e2be00640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 0x7f1e08069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.568+0000 7f1e2be00640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 0x7f1e08069fe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0af000 tx=0x7f1e1c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.572+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1e140555d0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.748+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1e2c04f4f0 con 0x7f1e2c0cde40 2024-03-21T08:42:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.748+0000 7f1e28c00640 1 -- 172.21.15.28:0/3928960219 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1e14059670 con 0x7f1e2c0cde40 2024-03-21T08:42:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 msgr2=0x7f1e08069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 0x7f1e08069fe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0af000 tx=0x7f1e1c0a7070 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 msgr2=0x7f1e2c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 0x7f1e2c15ad50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400c3c0 tx=0x7f1e1400c880 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 shutdown_connections 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c15cbb0 0x7f1e2c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e08067b20 0x7f1e08069fe0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e2c0cde40 0x7f1e2c15ad50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 --2- 172.21.15.28:0/3928960219 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e2c071a80 0x7f1e2c1364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 >> 172.21.15.28:0/3928960219 conn(0x7f1e2c0674f0 msgr2=0x7f1e2c0ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 shutdown_connections 2024-03-21T08:42:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.752+0000 7f1e33200640 1 -- 172.21.15.28:0/3928960219 wait complete. 2024-03-21T08:42:07.773 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:07.773 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:07.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:07.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:42:07.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:42:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.960+0000 7fe93b200640 1 Processor -- start 2024-03-21T08:42:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.960+0000 7fe93b200640 1 -- start start 2024-03-21T08:42:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe93b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe93415e610 0x7fe93415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe93b200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe9340b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe93b200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93415ebf0 con 0x7fe9340ce490 2024-03-21T08:42:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe93b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93415eed0 con 0x7fe93415e610 2024-03-21T08:42:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe93b200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93415f1b0 con 0x7fe9340b4380 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe93415e610 0x7fe93415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe93415e610 0x7fe93415ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54018/0 (socket says 172.21.15.28:54018) 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 -- 172.21.15.28:0/283478416 learned_addr learned my addr 172.21.15.28:0/283478416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe939200640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe9340b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 -- 172.21.15.28:0/283478416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 msgr2=0x7fe9340b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe9340b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 -- 172.21.15.28:0/283478416 >> v1:172.21.15.151:6789/0 conn(0x7fe9340ce490 legacy=0x7fe9340ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe92be00640 1 -- 172.21.15.28:0/283478416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93415f490 con 0x7fe93415e610 2024-03-21T08:42:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.964+0000 7fe933400640 1 -- 172.21.15.28:0/283478416 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005745305 0 0) 0x7fe93415ebf0 con 0x7fe9340ce490 2024-03-21T08:42:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe92be00640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415ba70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009870 tx=0x7fe91c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb20f87bf6e9791 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe933400640 1 -- 172.21.15.28:0/283478416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0b0070 con 0x7fe93415e610 2024-03-21T08:42:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe933400640 1 -- 172.21.15.28:0/283478416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c0ab600 con 0x7fe93415e610 2024-03-21T08:42:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe933400640 1 -- 172.21.15.28:0/283478416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0ab9d0 con 0x7fe93415e610 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- 172.21.15.28:0/283478416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 msgr2=0x7fe93415ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415ba70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009870 tx=0x7fe91c01c4d0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- 172.21.15.28:0/283478416 shutdown_connections 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe9340b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 --2- 172.21.15.28:0/283478416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415ba70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- 172.21.15.28:0/283478416 >> 172.21.15.28:0/283478416 conn(0x7fe9340cc230 msgr2=0x7fe9340cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- 172.21.15.28:0/283478416 shutdown_connections 2024-03-21T08:42:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- 172.21.15.28:0/283478416 wait complete. 2024-03-21T08:42:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 Processor -- start 2024-03-21T08:42:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.968+0000 7fe93b200640 1 -- start start 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe93406d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9340ce490 0x7fe9340afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe934167d40 con 0x7fe93415e610 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe934167bc0 con 0x7fe9340ce490 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe934167ec0 con 0x7fe9340b4380 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54022/0 (socket says 172.21.15.28:54022) 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 -- 172.21.15.28:0/4068325058 learned_addr learned my addr 172.21.15.28:0/4068325058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe933e00640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe93406d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe92be00640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9340ce490 0x7fe9340afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 -- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 msgr2=0x7fe93406d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe93406d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 -- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9340ce490 msgr2=0x7fe9340afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9340ce490 0x7fe9340afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 -- 172.21.15.28:0/4068325058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93414bbc0 con 0x7fe93415e610 2024-03-21T08:42:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe939200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00efc0 tx=0x7fe92c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c018020 con 0x7fe93415e610 2024-03-21T08:42:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c014480 con 0x7fe93415e610 2024-03-21T08:42:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c014870 con 0x7fe93415e610 2024-03-21T08:42:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93414d480 con 0x7fe93415e610 2024-03-21T08:42:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93414bea0 con 0x7fe93415e610 2024-03-21T08:42:07.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93404f410 con 0x7fe93415e610 2024-03-21T08:42:07.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.972+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe92c01b020 con 0x7fe93415e610 2024-03-21T08:42:07.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.976+0000 7fe930c00640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 0x7fe910069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.976+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fe92c086c80 con 0x7fe93415e610 2024-03-21T08:42:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.976+0000 7fe933e00640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 0x7fe910069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.976+0000 7fe933e00640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 0x7fe910069fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe924000f70 tx=0x7fe924009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:07.980+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe92c022070 con 0x7fe93415e610 2024-03-21T08:42:08.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.144+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe934072c10 con 0x7fe93415e610 2024-03-21T08:42:08.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe930c00640 1 -- 172.21.15.28:0/4068325058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7fe92c010040 con 0x7fe93415e610 2024-03-21T08:42:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:08.149 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 msgr2=0x7fe910069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 0x7fe910069fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe924000f70 tx=0x7fe924009040 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 msgr2=0x7fe93415dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00efc0 tx=0x7fe92c00c490 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 shutdown_connections 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93415e610 0x7fe93415dd10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe9340ce490 0x7fe9340afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe910067b20 0x7fe910069fe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 --2- 172.21.15.28:0/4068325058 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe9340b4380 0x7fe93406d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 >> 172.21.15.28:0/4068325058 conn(0x7fe9340cc230 msgr2=0x7fe934164cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 shutdown_connections 2024-03-21T08:42:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.148+0000 7fe93b200640 1 -- 172.21.15.28:0/4068325058 wait complete. 2024-03-21T08:42:08.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:08.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 Processor -- start 2024-03-21T08:42:08.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 -- start start 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb5c815df10 0x7fb5c806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5c806d1d0 0x7fb5c80d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c80b48a0 con 0x7fb5c80d0e00 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c806d630 con 0x7fb5c806d1d0 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5ce600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c80cc0a0 con 0x7fb5c815df10 2024-03-21T08:42:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5c806d1d0 0x7fb5c80d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5c806d1d0 0x7fb5c80d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54032/0 (socket says 172.21.15.28:54032) 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 -- 172.21.15.28:0/3572646477 learned_addr learned my addr 172.21.15.28:0/3572646477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5c7400640 1 --2- 172.21.15.28:0/3572646477 >> v2:172.21.15.151:3300/0 conn(0x7fb5c815df10 0x7fb5c806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 -- 172.21.15.28:0/3572646477 >> v2:172.21.15.151:3300/0 conn(0x7fb5c815df10 msgr2=0x7fb5c806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 --2- 172.21.15.28:0/3572646477 >> v2:172.21.15.151:3300/0 conn(0x7fb5c815df10 0x7fb5c806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 -- 172.21.15.28:0/3572646477 >> v1:172.21.15.28:6789/0 conn(0x7fb5c80d0e00 legacy=0x7fb5c8099bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.316+0000 7fb5bea00640 1 -- 172.21.15.28:0/3572646477 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80d1620 con 0x7fb5c806d1d0 2024-03-21T08:42:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5bea00640 1 --2- 172.21.15.28:0/3572646477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c80d08c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8072650 tx=0x7fb5b001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0c2644511ade0c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5c6a00640 1 -- 172.21.15.28:0/3572646477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00b0070 con 0x7fb5c806d1d0 2024-03-21T08:42:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5c6a00640 1 -- 172.21.15.28:0/3572646477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b00ab600 con 0x7fb5c806d1d0 2024-03-21T08:42:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5c6a00640 1 -- 172.21.15.28:0/3572646477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00ab9d0 con 0x7fb5c806d1d0 2024-03-21T08:42:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- 172.21.15.28:0/3572646477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 msgr2=0x7fb5c80d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3572646477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c80d08c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8072650 tx=0x7fb5b001c490 comp rx=0 tx=0).stop 2024-03-21T08:42:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- 172.21.15.28:0/3572646477 shutdown_connections 2024-03-21T08:42:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3572646477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c80d08c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3572646477 >> v2:172.21.15.151:3300/0 conn(0x7fb5c815df10 0x7fb5c806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- 172.21.15.28:0/3572646477 >> 172.21.15.28:0/3572646477 conn(0x7fb5c80b3540 msgr2=0x7fb5c80b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- 172.21.15.28:0/3572646477 shutdown_connections 2024-03-21T08:42:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- 172.21.15.28:0/3572646477 wait complete. 2024-03-21T08:42:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 Processor -- start 2024-03-21T08:42:08.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.320+0000 7fb5ce600640 1 -- start start 2024-03-21T08:42:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5c80d0e00 0x7fb5c80c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 0x7fb5c80c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80d1390 con 0x7fb5c806d1d0 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80d1210 con 0x7fb5c815df10 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c816a9a0 con 0x7fb5c80d0e00 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54038/0 (socket says 172.21.15.28:54038) 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 -- 172.21.15.28:0/3887161015 learned_addr learned my addr 172.21.15.28:0/3887161015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5bea00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5c80d0e00 0x7fb5c80c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7e00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 0x7fb5c80c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 -- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5c80d0e00 msgr2=0x7fb5c80c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5c80d0e00 0x7fb5c80c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 -- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 msgr2=0x7fb5c80c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 0x7fb5c80c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 -- 172.21.15.28:0/3887161015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c815f330 con 0x7fb5c806d1d0 2024-03-21T08:42:08.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7e00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 0x7fb5c80c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:08.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5c7400640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e960 tx=0x7fb5b800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8017070 con 0x7fb5c806d1d0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b800ca00 con 0x7fb5c806d1d0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c815f050 con 0x7fb5c806d1d0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b800cdb0 con 0x7fb5c806d1d0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c815f920 con 0x7fb5c806d1d0 2024-03-21T08:42:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.324+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c804f4f0 con 0x7fb5c806d1d0 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.328+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb5b801a070 con 0x7fb5c806d1d0 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.328+0000 7fb5bfe00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 0x7fb5a4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.328+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fb5b80862d0 con 0x7fb5c806d1d0 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.328+0000 7fb5bea00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 0x7fb5a4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.328+0000 7fb5bea00640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 0x7fb5a4069fe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00ad000 tx=0x7fb5b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.332+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb5b8056270 con 0x7fb5c806d1d0 2024-03-21T08:42:08.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.496+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5c80cca50 con 0x7fb5c806d1d0 2024-03-21T08:42:08.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.496+0000 7fb5bfe00640 1 -- 172.21.15.28:0/3887161015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5b805a310 con 0x7fb5c806d1d0 2024-03-21T08:42:08.499 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 msgr2=0x7fb5a4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 0x7fb5a4069fe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00ad000 tx=0x7fb5b0008040 comp rx=0 tx=0).stop 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 msgr2=0x7fb5c81568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e960 tx=0x7fb5b800ee20 comp rx=0 tx=0).stop 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 shutdown_connections 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5c815df10 0x7fb5c80c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb5a4067b20 0x7fb5a4069fe0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5c80d0e00 0x7fb5c80c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 --2- 172.21.15.28:0/3887161015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5c806d1d0 0x7fb5c81568b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 >> 172.21.15.28:0/3887161015 conn(0x7fb5c80b3540 msgr2=0x7fb5c80b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 shutdown_connections 2024-03-21T08:42:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.500+0000 7fb5ce600640 1 -- 172.21.15.28:0/3887161015 wait complete. 2024-03-21T08:42:08.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.660+0000 7f4da2800640 1 Processor -- start 2024-03-21T08:42:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.660+0000 7f4da2800640 1 -- start start 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4da2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d9c15ee40 0x7f4d9c0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4da2800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4da2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c06dd60 con 0x7f4d9c072360 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4da2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c072b10 con 0x7f4d9c15ee40 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4da2800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c072df0 con 0x7f4d9c0b39c0 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d9c15ee40 0x7f4d9c0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d9c072360 0x7f4d9c161900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55690/0 (socket says 172.21.15.28:55690) 2024-03-21T08:42:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9be00640 1 -- 172.21.15.28:0/3025804117 learned_addr learned my addr 172.21.15.28:0/3025804117 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d92a00640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9b400640 1 -- 172.21.15.28:0/3025804117 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 msgr2=0x7f4d9c071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9b400640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9b400640 1 -- 172.21.15.28:0/3025804117 >> v1:172.21.15.28:6789/0 conn(0x7f4d9c072360 legacy=0x7f4d9c161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9b400640 1 -- 172.21.15.28:0/3025804117 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c15f220 con 0x7f4d9c15ee40 2024-03-21T08:42:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.664+0000 7f4d9aa00640 1 -- 172.21.15.28:0/3025804117 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279809913 0 0) 0x7f4d9c06dd60 con 0x7f4d9c072360 2024-03-21T08:42:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4d9b400640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0b3480 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0097c0 tx=0x7f4d8c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc71890331f6a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4d9aa00640 1 -- 172.21.15.28:0/3025804117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0b0070 con 0x7f4d9c15ee40 2024-03-21T08:42:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4d9aa00640 1 -- 172.21.15.28:0/3025804117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c0ab600 con 0x7f4d9c15ee40 2024-03-21T08:42:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4d9aa00640 1 -- 172.21.15.28:0/3025804117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0ab9d0 con 0x7f4d9c15ee40 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- 172.21.15.28:0/3025804117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 msgr2=0x7f4d9c0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0b3480 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0097c0 tx=0x7f4d8c01c410 comp rx=0 tx=0).stop 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- 172.21.15.28:0/3025804117 shutdown_connections 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 --2- 172.21.15.28:0/3025804117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0b3480 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- 172.21.15.28:0/3025804117 >> 172.21.15.28:0/3025804117 conn(0x7f4d9c0b1a90 msgr2=0x7f4d9c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- 172.21.15.28:0/3025804117 shutdown_connections 2024-03-21T08:42:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- 172.21.15.28:0/3025804117 wait complete. 2024-03-21T08:42:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 Processor -- start 2024-03-21T08:42:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.668+0000 7f4da2800640 1 -- start start 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d9c072360 0x7f4d9c14d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c13e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c16ae20 con 0x7f4d9c15ee40 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c16aca0 con 0x7f4d9c072360 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9c16afa0 con 0x7f4d9c0b39c0 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54056/0 (socket says 172.21.15.28:54056) 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 -- 172.21.15.28:0/3131089256 learned_addr learned my addr 172.21.15.28:0/3131089256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9b400640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d9c072360 0x7f4d9c14d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d92a00640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c13e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 -- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 msgr2=0x7f4d9c13e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c13e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 -- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d9c072360 msgr2=0x7f4d9c14d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d9c072360 0x7f4d9c14d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 -- 172.21.15.28:0/3131089256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0adf20 con 0x7f4d9c15ee40 2024-03-21T08:42:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d9be00640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400c700 tx=0x7f4d9400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94018070 con 0x7f4d9c15ee40 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9400ce10 con 0x7f4d9c15ee40 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9400a860 con 0x7f4d9c15ee40 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c14d860 con 0x7f4d9c15ee40 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c14e330 con 0x7f4d9c15ee40 2024-03-21T08:42:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c09e0f0 con 0x7f4d9c15ee40 2024-03-21T08:42:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.672+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4d9400aa00 con 0x7f4d9c15ee40 2024-03-21T08:42:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.676+0000 7f4d93e00640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 0x7f4d78069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.676+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f4d94085930 con 0x7f4d9c15ee40 2024-03-21T08:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.676+0000 7f4d9b400640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 0x7f4d78069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.676+0000 7f4d9b400640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 0x7f4d78069fe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0ad000 tx=0x7f4d8c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.680+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4d940557b0 con 0x7f4d9c15ee40 2024-03-21T08:42:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.844+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d9c04f4f0 con 0x7f4d9c15ee40 2024-03-21T08:42:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.844+0000 7f4d93e00640 1 -- 172.21.15.28:0/3131089256 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4d94059850 con 0x7f4d9c15ee40 2024-03-21T08:42:08.846 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 msgr2=0x7f4d78069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 0x7f4d78069fe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0ad000 tx=0x7f4d8c0a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 msgr2=0x7f4d9c0ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400c700 tx=0x7f4d9400cbc0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 shutdown_connections 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9c15ee40 0x7f4d9c0ad9e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4d78067b20 0x7f4d78069fe0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4d9c0b39c0 0x7f4d9c13e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 --2- 172.21.15.28:0/3131089256 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4d9c072360 0x7f4d9c14d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 >> 172.21.15.28:0/3131089256 conn(0x7f4d9c0b1a90 msgr2=0x7f4d9c15d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 shutdown_connections 2024-03-21T08:42:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:08.848+0000 7f4da2800640 1 -- 172.21.15.28:0/3131089256 wait complete. 2024-03-21T08:42:08.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.008+0000 7fad33200640 1 Processor -- start 2024-03-21T08:42:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.008+0000 7fad33200640 1 -- start start 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad33200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad2c0d0320 0x7fad2c0d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad33200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fad2c15e190 0x7fad2c0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad33200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0d0c40 con 0x7fad2c0d0ee0 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad33200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c0b42a0 con 0x7fad2c0d0320 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad33200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c06bd40 con 0x7fad2c15e190 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad2c0d0320 0x7fad2c0d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad2c0d0320 0x7fad2c0d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54072/0 (socket says 172.21.15.28:54072) 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 -- 172.21.15.28:0/2940664744 learned_addr learned my addr 172.21.15.28:0/2940664744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad23e00640 1 --2- 172.21.15.28:0/2940664744 >> v2:172.21.15.151:3300/0 conn(0x7fad2c15e190 0x7fad2c0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 -- 172.21.15.28:0/2940664744 >> v2:172.21.15.151:3300/0 conn(0x7fad2c15e190 msgr2=0x7fad2c0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 --2- 172.21.15.28:0/2940664744 >> v2:172.21.15.151:3300/0 conn(0x7fad2c15e190 0x7fad2c0b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 -- 172.21.15.28:0/2940664744 >> v1:172.21.15.151:6789/0 conn(0x7fad2c0d0ee0 legacy=0x7fad2c0af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.012+0000 7fad2be00640 1 -- 172.21.15.28:0/2940664744 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c06c020 con 0x7fad2c0d0320 2024-03-21T08:42:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad2be00640 1 --2- 172.21.15.28:0/2940664744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c0d0700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1619b0 tx=0x7fad1c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9143bde3aa289a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad2b400640 1 -- 172.21.15.28:0/2940664744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0ab430 con 0x7fad2c0d0320 2024-03-21T08:42:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad2b400640 1 -- 172.21.15.28:0/2940664744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c0b0070 con 0x7fad2c0d0320 2024-03-21T08:42:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad2b400640 1 -- 172.21.15.28:0/2940664744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0ab7f0 con 0x7fad2c0d0320 2024-03-21T08:42:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- 172.21.15.28:0/2940664744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 msgr2=0x7fad2c0d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- 172.21.15.28:0/2940664744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c0d0700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1619b0 tx=0x7fad1c01c520 comp rx=0 tx=0).stop 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- 172.21.15.28:0/2940664744 shutdown_connections 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- 172.21.15.28:0/2940664744 >> v2:172.21.15.151:3300/0 conn(0x7fad2c15e190 0x7fad2c0b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- 172.21.15.28:0/2940664744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c0d0700 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- 172.21.15.28:0/2940664744 >> 172.21.15.28:0/2940664744 conn(0x7fad2c099a30 msgr2=0x7fad2c0b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- 172.21.15.28:0/2940664744 shutdown_connections 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- 172.21.15.28:0/2940664744 wait complete. 2024-03-21T08:42:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 Processor -- start 2024-03-21T08:42:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- start start 2024-03-21T08:42:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad2c0d0ee0 0x7fad2c15bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad2c15e190 0x7fad2c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c16ae50 con 0x7fad2c0d0320 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c16acd0 con 0x7fad2c15e190 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.016+0000 7fad33200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c16afd0 con 0x7fad2c0d0ee0 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54082/0 (socket says 172.21.15.28:54082) 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 -- 172.21.15.28:0/98862338 learned_addr learned my addr 172.21.15.28:0/98862338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad31200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad2c15e190 0x7fad2c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad23e00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad2c0d0ee0 0x7fad2c15bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 -- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad2c0d0ee0 msgr2=0x7fad2c15bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad2c0d0ee0 0x7fad2c15bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 -- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad2c15e190 msgr2=0x7fad2c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad2c15e190 0x7fad2c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 -- 172.21.15.28:0/98862338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c15ecd0 con 0x7fad2c0d0320 2024-03-21T08:42:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad2be00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fad1c01a150 tx=0x7fad1c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0be5f0 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c0be790 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0beb60 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c15efb0 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c152e10 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c0700a0 con 0x7fad2c0d0320 2024-03-21T08:42:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fad1c0b43f0 con 0x7fad2c0d0320 2024-03-21T08:42:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.020+0000 7fad28c00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 0x7fad0806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.024+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fad1c0a8030 con 0x7fad2c0d0320 2024-03-21T08:42:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.024+0000 7fad23e00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 0x7fad0806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.024+0000 7fad23e00640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 0x7fad0806a010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fad2c156c40 tx=0x7fad14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.024+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fad1c016110 con 0x7fad2c0d0320 2024-03-21T08:42:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.188+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad2c04f4f0 con 0x7fad2c0d0320 2024-03-21T08:42:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.188+0000 7fad28c00640 1 -- 172.21.15.28:0/98862338 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad1c0f03c0 con 0x7fad2c0d0320 2024-03-21T08:42:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 msgr2=0x7fad0806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 0x7fad0806a010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fad2c156c40 tx=0x7fad14009190 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 msgr2=0x7fad2c143d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fad1c01a150 tx=0x7fad1c01ad60 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 shutdown_connections 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad2c15e190 0x7fad2c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad08067b50 0x7fad0806a010 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad2c0d0ee0 0x7fad2c15bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 --2- 172.21.15.28:0/98862338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2c0d0320 0x7fad2c143d40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 >> 172.21.15.28:0/98862338 conn(0x7fad2c099a30 msgr2=0x7fad2c06f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 shutdown_connections 2024-03-21T08:42:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.192+0000 7fad33200640 1 -- 172.21.15.28:0/98862338 wait complete. 2024-03-21T08:42:09.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.352+0000 7f06a3200640 1 Processor -- start 2024-03-21T08:42:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.352+0000 7f06a3200640 1 -- start start 2024-03-21T08:42:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f069c06b880 0x7f069c0cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0d0760 con 0x7f069c0ccb70 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a3200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0cfec0 con 0x7f069c15e020 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c0ce7b0 con 0x7f069c06b880 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f069c06b880 0x7f069c0cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f0693400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f069c0ccb70 0x7f069c06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55706/0 (socket says 172.21.15.28:55706) 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f0693400640 1 -- 172.21.15.28:0/4292667509 learned_addr learned my addr 172.21.15.28:0/4292667509 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 --2- 172.21.15.28:0/4292667509 >> v2:172.21.15.28:3300/0 conn(0x7f069c06b880 0x7f069c0cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54094/0 (socket says 172.21.15.28:54094) 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 -- 172.21.15.28:0/4292667509 >> v1:172.21.15.151:6789/0 conn(0x7f069c15e020 legacy=0x7f069c0cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 -- 172.21.15.28:0/4292667509 >> v1:172.21.15.28:6789/0 conn(0x7f069c0ccb70 legacy=0x7f069c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 -- 172.21.15.28:0/4292667509 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0d01a0 con 0x7f069c06b880 2024-03-21T08:42:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.356+0000 7f06a1200640 1 --2- 172.21.15.28:0/4292667509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c0cd000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f06940097f0 tx=0x7f069401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53464f0a8cc6948d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069b400640 1 -- 172.21.15.28:0/4292667509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06940b0070 con 0x7f069c06b880 2024-03-21T08:42:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069b400640 1 -- 172.21.15.28:0/4292667509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06940ab600 con 0x7f069c06b880 2024-03-21T08:42:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069b400640 1 -- 172.21.15.28:0/4292667509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06940ab9d0 con 0x7f069c06b880 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- 172.21.15.28:0/4292667509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 msgr2=0x7f069c0cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 --2- 172.21.15.28:0/4292667509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c0cd000 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f06940097f0 tx=0x7f069401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- 172.21.15.28:0/4292667509 shutdown_connections 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 --2- 172.21.15.28:0/4292667509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c0cd000 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- 172.21.15.28:0/4292667509 >> 172.21.15.28:0/4292667509 conn(0x7f069c099a30 msgr2=0x7f069c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- 172.21.15.28:0/4292667509 shutdown_connections 2024-03-21T08:42:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- 172.21.15.28:0/4292667509 wait complete. 2024-03-21T08:42:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 Processor -- start 2024-03-21T08:42:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- start start 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f069c0ccb70 0x7f069c0c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f069c15e020 0x7f069c0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c15e790 con 0x7f069c06b880 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c15f6a0 con 0x7f069c15e020 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a3200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c16bb30 con 0x7f069c0ccb70 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54108/0 (socket says 172.21.15.28:54108) 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f069be00640 1 -- 172.21.15.28:0/364078787 learned_addr learned my addr 172.21.15.28:0/364078787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f06a1200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f069c15e020 0x7f069c0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.360+0000 7f0693400640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f069c0ccb70 0x7f069c0c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 -- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f069c0ccb70 msgr2=0x7f069c0c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f069c0ccb70 0x7f069c0c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 -- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f069c15e020 msgr2=0x7f069c0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f069c15e020 0x7f069c0c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 -- 172.21.15.28:0/364078787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0c9960 con 0x7f069c06b880 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f069be00640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f068c00c760 tx=0x7f068c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c003ba0 con 0x7f069c06b880 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068c003d60 con 0x7f069c06b880 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c00aa30 con 0x7f069c06b880 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c0c9c40 con 0x7f069c06b880 2024-03-21T08:42:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c0ca200 con 0x7f069c06b880 2024-03-21T08:42:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c04f4f0 con 0x7f069c06b880 2024-03-21T08:42:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f068c00abd0 con 0x7f069c06b880 2024-03-21T08:42:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 0x7f0678069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f068c0860c0 con 0x7f069c06b880 2024-03-21T08:42:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.364+0000 7f0693400640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 0x7f0678069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.368+0000 7f0693400640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 0x7f0678069fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f069c157fc0 tx=0x7f0684009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.368+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f068c056060 con 0x7f069c06b880 2024-03-21T08:42:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.532+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f069c0b1df0 con 0x7f069c06b880 2024-03-21T08:42:09.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.532+0000 7f0698c00640 1 -- 172.21.15.28:0/364078787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f068c05a100 con 0x7f069c06b880 2024-03-21T08:42:09.534 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 msgr2=0x7f0678069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 0x7f0678069fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f069c157fc0 tx=0x7f0684009040 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 msgr2=0x7f069c14a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f068c00c760 tx=0x7f068c00cc20 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 shutdown_connections 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f069c15e020 0x7f069c0c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0678067b20 0x7f0678069fe0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f069c0ccb70 0x7f069c0c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 --2- 172.21.15.28:0/364078787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c06b880 0x7f069c14a220 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 >> 172.21.15.28:0/364078787 conn(0x7f069c099a30 msgr2=0x7f069c070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 shutdown_connections 2024-03-21T08:42:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.536+0000 7f06a3200640 1 -- 172.21.15.28:0/364078787 wait complete. 2024-03-21T08:42:09.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 Processor -- start 2024-03-21T08:42:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 -- start start 2024-03-21T08:42:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 0x7fc940071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc9400cd300 0x7fc940071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94015bf90 con 0x7fc9400b4800 2024-03-21T08:42:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94015b1d0 con 0x7fc9400cd300 2024-03-21T08:42:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc947c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94015b4b0 con 0x7fc9400b4420 2024-03-21T08:42:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc937e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc9400b4800 0x7fc9400ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55722/0 (socket says 172.21.15.28:55722) 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc937e00640 1 -- 172.21.15.28:0/3475132903 learned_addr learned my addr 172.21.15.28:0/3475132903 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc945200640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 0x7fc940071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.696+0000 7fc945c00640 1 --2- 172.21.15.28:0/3475132903 >> v2:172.21.15.151:3300/0 conn(0x7fc9400cd300 0x7fc940071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc93fe00640 1 -- 172.21.15.28:0/3475132903 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995550685 0 0) 0x7fc94015bf90 con 0x7fc9400b4800 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc93fe00640 1 -- 172.21.15.28:0/3475132903 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc924005740 con 0x7fc9400b4800 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945c00640 1 -- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 msgr2=0x7fc940071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945c00640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 0x7fc940071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945c00640 1 -- 172.21.15.28:0/3475132903 >> v1:172.21.15.28:6789/0 conn(0x7fc9400b4800 legacy=0x7fc9400ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945c00640 1 -- 172.21.15.28:0/3475132903 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94015b790 con 0x7fc9400cd300 2024-03-21T08:42:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945200640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 0x7fc940071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc945c00640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9400cd300 0x7fc940071fc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc938009d90 tx=0x7fc93801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196ca315f0dbc217 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc93fe00640 1 -- 172.21.15.28:0/3475132903 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9380b2070 con 0x7fc9400cd300 2024-03-21T08:42:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc93fe00640 1 -- 172.21.15.28:0/3475132903 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9380a5800 con 0x7fc9400cd300 2024-03-21T08:42:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.700+0000 7fc93fe00640 1 -- 172.21.15.28:0/3475132903 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9380a5bb0 con 0x7fc9400cd300 2024-03-21T08:42:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9400cd300 msgr2=0x7fc940071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9400cd300 0x7fc940071fc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc938009d90 tx=0x7fc93801c8b0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- 172.21.15.28:0/3475132903 shutdown_connections 2024-03-21T08:42:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc9400cd300 0x7fc940071fc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- 172.21.15.28:0/3475132903 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400b4420 0x7fc940071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- 172.21.15.28:0/3475132903 >> 172.21.15.28:0/3475132903 conn(0x7fc9400674f0 msgr2=0x7fc94006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- 172.21.15.28:0/3475132903 shutdown_connections 2024-03-21T08:42:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- 172.21.15.28:0/3475132903 wait complete. 2024-03-21T08:42:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 Processor -- start 2024-03-21T08:42:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- start start 2024-03-21T08:42:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 0x7fc9400cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc940158100 0x7fc940158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94006e680 con 0x7fc9400b4420 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94006e500 con 0x7fc940158100 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc947c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94006e800 con 0x7fc9400cd300 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54122/0 (socket says 172.21.15.28:54122) 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 -- 172.21.15.28:0/1097368917 learned_addr learned my addr 172.21.15.28:0/1097368917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc937e00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 0x7fc9400cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc940158100 0x7fc940158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 -- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 msgr2=0x7fc9400cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 0x7fc9400cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 -- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc940158100 msgr2=0x7fc940158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc940158100 0x7fc940158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 -- 172.21.15.28:0/1097368917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc940157060 con 0x7fc9400b4420 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc937e00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 0x7fc9400cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.704+0000 7fc945200640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc93000e980 tx=0x7fc93000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930017070 con 0x7fc9400b4420 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93000ca00 con 0x7fc9400b4420 2024-03-21T08:42:09.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93000cdd0 con 0x7fc9400b4420 2024-03-21T08:42:09.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc940157340 con 0x7fc9400b4420 2024-03-21T08:42:09.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc940157900 con 0x7fc9400b4420 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc940073430 con 0x7fc9400b4420 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc93001a070 con 0x7fc9400b4420 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 0x7fc91c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fc9300865a0 con 0x7fc9400b4420 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc937e00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 0x7fc91c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.708+0000 7fc937e00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 0x7fc91c069fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc9400cbc30 tx=0x7fc928009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:09.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.712+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc9300564b0 con 0x7fc9400b4420 2024-03-21T08:42:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.888+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc9401478f0 con 0x7fc9400b4420 2024-03-21T08:42:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.888+0000 7fc93d600640 1 -- 172.21.15.28:0/1097368917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc93005a550 con 0x7fc9400b4420 2024-03-21T08:42:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 msgr2=0x7fc91c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 0x7fc91c069fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc9400cbc30 tx=0x7fc928009040 comp rx=0 tx=0).stop 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 msgr2=0x7fc9400caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc93000e980 tx=0x7fc93000ee40 comp rx=0 tx=0).stop 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 shutdown_connections 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc940158100 0x7fc940158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc91c067b20 0x7fc91c069fe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc9400cd300 0x7fc9400cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 --2- 172.21.15.28:0/1097368917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9400b4420 0x7fc9400caf00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 >> 172.21.15.28:0/1097368917 conn(0x7fc9400674f0 msgr2=0x7fc94015db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 shutdown_connections 2024-03-21T08:42:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:09.892+0000 7fc947c00640 1 -- 172.21.15.28:0/1097368917 wait complete. 2024-03-21T08:42:09.912 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:42:09.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:09.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-21T08:42:09.933 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:09.933 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:42:09.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:09.933 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-21T08:42:09.984 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:09.984 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:09.993 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:09.994 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:10.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.180+0000 7f2d2c600640 1 Processor -- start 2024-03-21T08:42:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.180+0000 7f2d2c600640 1 -- start start 2024-03-21T08:42:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2c600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2d240aff70 0x7f2d2406b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2c600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240d0540 con 0x7f2d2406eb30 2024-03-21T08:42:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2c600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d240cd6e0 con 0x7f2d2415dd00 2024-03-21T08:42:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2c600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d240cd9c0 con 0x7f2d240aff70 2024-03-21T08:42:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d29200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2d240aff70 0x7f2d2406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2a600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2d2406eb30 0x7f2d24161a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55736/0 (socket says 172.21.15.28:55736) 2024-03-21T08:42:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d2a600640 1 -- 172.21.15.28:0/3314121778 learned_addr learned my addr 172.21.15.28:0/3314121778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:10.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d23e00640 1 -- 172.21.15.28:0/3314121778 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162263251 0 0) 0x7f2d240d0540 con 0x7f2d2406eb30 2024-03-21T08:42:10.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d29200640 1 -- 172.21.15.28:0/3314121778 >> v1:172.21.15.151:6789/0 conn(0x7f2d2415dd00 legacy=0x7f2d240cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d29200640 1 -- 172.21.15.28:0/3314121778 >> v1:172.21.15.28:6789/0 conn(0x7f2d2406eb30 legacy=0x7f2d24161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.184+0000 7f2d29200640 1 -- 172.21.15.28:0/3314121778 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240cdca0 con 0x7f2d240aff70 2024-03-21T08:42:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d29200640 1 --2- 172.21.15.28:0/3314121778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d240aff70 0x7f2d2406b170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2d100097c0 tx=0x7f2d1001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1e7fbfa43031ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d23e00640 1 -- 172.21.15.28:0/3314121778 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d100b2070 con 0x7f2d240aff70 2024-03-21T08:42:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d23e00640 1 -- 172.21.15.28:0/3314121778 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d100ad680 con 0x7f2d240aff70 2024-03-21T08:42:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d23e00640 1 -- 172.21.15.28:0/3314121778 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d100ada50 con 0x7f2d240aff70 2024-03-21T08:42:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- 172.21.15.28:0/3314121778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d240aff70 msgr2=0x7f2d2406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 --2- 172.21.15.28:0/3314121778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d240aff70 0x7f2d2406b170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2d100097c0 tx=0x7f2d1001c470 comp rx=0 tx=0).stop 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- 172.21.15.28:0/3314121778 shutdown_connections 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 --2- 172.21.15.28:0/3314121778 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d240aff70 0x7f2d2406b170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- 172.21.15.28:0/3314121778 >> 172.21.15.28:0/3314121778 conn(0x7f2d24099a30 msgr2=0x7f2d24099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- 172.21.15.28:0/3314121778 shutdown_connections 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- 172.21.15.28:0/3314121778 wait complete. 2024-03-21T08:42:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 Processor -- start 2024-03-21T08:42:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- start start 2024-03-21T08:42:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d2406eb30 0x7f2d24154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2d240aff70 0x7f2d2415edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2416be70 con 0x7f2d2415dd00 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2416bcf0 con 0x7f2d2406eb30 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.188+0000 7f2d2c600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2416bff0 con 0x7f2d240aff70 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54134/0 (socket says 172.21.15.28:54134) 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 -- 172.21.15.28:0/2894927666 learned_addr learned my addr 172.21.15.28:0/2894927666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d29200640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2d240aff70 0x7f2d2415edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d29c00640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d2406eb30 0x7f2d24154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 -- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2d240aff70 msgr2=0x7f2d2415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2d240aff70 0x7f2d2415edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 -- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d2406eb30 msgr2=0x7f2d24154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d2406eb30 0x7f2d24154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 -- 172.21.15.28:0/2894927666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2415f750 con 0x7f2d2415dd00 2024-03-21T08:42:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2a600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c7a0 tx=0x7f2d1c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c018070 con 0x7f2d2415dd00 2024-03-21T08:42:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00a640 con 0x7f2d2415dd00 2024-03-21T08:42:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00aa70 con 0x7f2d2415dd00 2024-03-21T08:42:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2415fa30 con 0x7f2d2415dd00 2024-03-21T08:42:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2415fff0 con 0x7f2d2415dd00 2024-03-21T08:42:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2404f410 con 0x7f2d2415dd00 2024-03-21T08:42:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2d1c00ac10 con 0x7f2d2415dd00 2024-03-21T08:42:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.192+0000 7f2d21600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 0x7f2d04069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.196+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2d1c085900 con 0x7f2d2415dd00 2024-03-21T08:42:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.196+0000 7f2d29c00640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 0x7f2d04069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.196+0000 7f2d29c00640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 0x7f2d04069fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2d18007fd0 tx=0x7f2d18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.200+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2d1c0558a0 con 0x7f2d2415dd00 2024-03-21T08:42:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d240cd4c0 con 0x7f2d2415dd00 2024-03-21T08:42:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d21600640 1 -- 172.21.15.28:0/2894927666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f2d1c059940 con 0x7f2d2415dd00 2024-03-21T08:42:10.365 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:10.366 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:10.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 msgr2=0x7f2d04069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 0x7f2d04069fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2d18007fd0 tx=0x7f2d18009040 comp rx=0 tx=0).stop 2024-03-21T08:42:10.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 msgr2=0x7f2d24160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c7a0 tx=0x7f2d1c00cc60 comp rx=0 tx=0).stop 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 shutdown_connections 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d2415dd00 0x7f2d24160ad0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2d04067b20 0x7f2d04069fe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2d240aff70 0x7f2d2415edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 --2- 172.21.15.28:0/2894927666 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2d2406eb30 0x7f2d24154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 >> 172.21.15.28:0/2894927666 conn(0x7f2d24099a30 msgr2=0x7f2d240cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 shutdown_connections 2024-03-21T08:42:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.364+0000 7f2d2c600640 1 -- 172.21.15.28:0/2894927666 wait complete. 2024-03-21T08:42:10.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:10.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:10.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.528+0000 7fde3c200640 1 Processor -- start 2024-03-21T08:42:10.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.528+0000 7fde3c200640 1 -- start start 2024-03-21T08:42:10.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3c200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde340b4800 0x7fde340ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3c200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde34072190 0x7fde340cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340cd670 con 0x7fde340b4420 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3c200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3415bf90 con 0x7fde34072190 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3c200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3415b1d0 con 0x7fde340b4800 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde39800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde340b4420 0x7fde34071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39206/0 (socket says 172.21.15.28:39206) 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde39800640 1 -- 172.21.15.28:0/3106879118 learned_addr learned my addr 172.21.15.28:0/3106879118 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3a200640 1 --2- 172.21.15.28:0/3106879118 >> v2:172.21.15.28:3300/0 conn(0x7fde34072190 0x7fde340cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde38e00640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde340b4800 0x7fde340ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3a200640 1 -- 172.21.15.28:0/3106879118 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde340b4800 msgr2=0x7fde340ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3a200640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde340b4800 0x7fde340ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3a200640 1 -- 172.21.15.28:0/3106879118 >> v1:172.21.15.28:6789/0 conn(0x7fde340b4420 legacy=0x7fde34071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde3a200640 1 -- 172.21.15.28:0/3106879118 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3415b4b0 con 0x7fde34072190 2024-03-21T08:42:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.532+0000 7fde33e00640 1 -- 172.21.15.28:0/3106879118 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869731494 0 0) 0x7fde340cd670 con 0x7fde340b4420 2024-03-21T08:42:10.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3a200640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde34072190 0x7fde340cd130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009d90 tx=0x7fde2c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a1eebf9dfe6d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde33e00640 1 -- 172.21.15.28:0/3106879118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0b2070 con 0x7fde34072190 2024-03-21T08:42:10.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde33e00640 1 -- 172.21.15.28:0/3106879118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2c0a5a00 con 0x7fde34072190 2024-03-21T08:42:10.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde33e00640 1 -- 172.21.15.28:0/3106879118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c0a5dd0 con 0x7fde34072190 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- 172.21.15.28:0/3106879118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde34072190 msgr2=0x7fde340cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde34072190 0x7fde340cd130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009d90 tx=0x7fde2c01c940 comp rx=0 tx=0).stop 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- 172.21.15.28:0/3106879118 shutdown_connections 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde34072190 0x7fde340cd130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- 172.21.15.28:0/3106879118 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde340b4800 0x7fde340ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- 172.21.15.28:0/3106879118 >> 172.21.15.28:0/3106879118 conn(0x7fde340674f0 msgr2=0x7fde3406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- 172.21.15.28:0/3106879118 shutdown_connections 2024-03-21T08:42:10.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- 172.21.15.28:0/3106879118 wait complete. 2024-03-21T08:42:10.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 Processor -- start 2024-03-21T08:42:10.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- start start 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 0x7fde3413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde34155d80 0x7fde34156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde340b3460 con 0x7fde340b4420 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde340b32e0 con 0x7fde34155d80 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3c200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde340b35e0 con 0x7fde34072190 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde38e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde38e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42578/0 (socket says 172.21.15.28:42578) 2024-03-21T08:42:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde39800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 0x7fde3413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde38e00640 1 -- 172.21.15.28:0/3577175429 learned_addr learned my addr 172.21.15.28:0/3577175429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.536+0000 7fde3a200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde34155d80 0x7fde34156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 -- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde34155d80 msgr2=0x7fde34156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde34155d80 0x7fde34156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 -- 172.21.15.28:0/3577175429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 msgr2=0x7fde34155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 -- 172.21.15.28:0/3577175429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34161fe0 con 0x7fde34072190 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde38e00640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde39800640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 0x7fde3413e370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c430 tx=0x7fde2800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde280074e0 con 0x7fde34072190 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde280047f0 con 0x7fde34072190 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde341588c0 con 0x7fde34072190 2024-03-21T08:42:10.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde28004be0 con 0x7fde34072190 2024-03-21T08:42:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde34158e80 con 0x7fde34072190 2024-03-21T08:42:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde34073430 con 0x7fde34072190 2024-03-21T08:42:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fde28011070 con 0x7fde34072190 2024-03-21T08:42:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 0x7fde14069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.540+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fde28085ec0 con 0x7fde34072190 2024-03-21T08:42:10.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.544+0000 7fde38e00640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 0x7fde14069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.544+0000 7fde38e00640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 0x7fde14069fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fde341655a0 tx=0x7fde20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.552+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fde28055e90 con 0x7fde34072190 2024-03-21T08:42:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.712+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde3406e990 con 0x7fde34072190 2024-03-21T08:42:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.712+0000 7fde31600640 1 -- 172.21.15.28:0/3577175429 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde28059f30 con 0x7fde34072190 2024-03-21T08:42:10.716 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 msgr2=0x7fde14069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 0x7fde14069fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fde341655a0 tx=0x7fde20009190 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 msgr2=0x7fde3413e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 0x7fde3413e370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c430 tx=0x7fde2800c8f0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 shutdown_connections 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fde34155d80 0x7fde34156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde340b4420 0x7fde34155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fde14067b20 0x7fde14069fe0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 --2- 172.21.15.28:0/3577175429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fde34072190 0x7fde3413e370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 >> 172.21.15.28:0/3577175429 conn(0x7fde340674f0 msgr2=0x7fde34159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 shutdown_connections 2024-03-21T08:42:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.716+0000 7fde3c200640 1 -- 172.21.15.28:0/3577175429 wait complete. 2024-03-21T08:42:10.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 Processor -- start 2024-03-21T08:42:10.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 -- start start 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11f815e190 0x7f11f80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80b42a0 con 0x7f11f80d0320 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f806bd40 con 0x7f11f80d0ee0 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11fec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f806c020 con 0x7f11f815e190 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f11f80d0320 0x7f11f80d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39216/0 (socket says 172.21.15.28:39216) 2024-03-21T08:42:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7e00640 1 -- 172.21.15.28:0/681091383 learned_addr learned my addr 172.21.15.28:0/681091383 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7400640 1 --2- 172.21.15.28:0/681091383 >> v2:172.21.15.28:3300/0 conn(0x7f11f815e190 0x7f11f80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435968405 0 0) 0x7f11f80b42a0 con 0x7f11f80d0320 2024-03-21T08:42:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc003530 con 0x7f11f80d0320 2024-03-21T08:42:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785181454 0 0) 0x7f11f806bd40 con 0x7f11f80d0ee0 2024-03-21T08:42:10.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f80b42a0 con 0x7f11f80d0ee0 2024-03-21T08:42:10.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7400640 1 -- 172.21.15.28:0/681091383 >> v1:172.21.15.151:6789/0 conn(0x7f11f80d0ee0 legacy=0x7f11f80ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7400640 1 -- 172.21.15.28:0/681091383 >> v1:172.21.15.28:6789/0 conn(0x7f11f80d0320 legacy=0x7f11f80d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.880+0000 7f11f7400640 1 -- 172.21.15.28:0/681091383 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f806c300 con 0x7f11f815e190 2024-03-21T08:42:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11f7400640 1 --2- 172.21.15.28:0/681091383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f80b3d60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f11e40097c0 tx=0x7f11e401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591ff18f9bd9e5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e40b0070 con 0x7f11f815e190 2024-03-21T08:42:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e40ab600 con 0x7f11f815e190 2024-03-21T08:42:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11f6a00640 1 -- 172.21.15.28:0/681091383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e40ab9d0 con 0x7f11f815e190 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 -- 172.21.15.28:0/681091383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 msgr2=0x7f11f80b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 --2- 172.21.15.28:0/681091383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f80b3d60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f11e40097c0 tx=0x7f11e401c410 comp rx=0 tx=0).stop 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 -- 172.21.15.28:0/681091383 shutdown_connections 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 --2- 172.21.15.28:0/681091383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f80b3d60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 -- 172.21.15.28:0/681091383 >> 172.21.15.28:0/681091383 conn(0x7f11f8099a30 msgr2=0x7f11f80b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 -- 172.21.15.28:0/681091383 shutdown_connections 2024-03-21T08:42:10.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.884+0000 7f11fec00640 1 -- 172.21.15.28:0/681091383 wait complete. 2024-03-21T08:42:10.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 Processor -- start 2024-03-21T08:42:10.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- start start 2024-03-21T08:42:10.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11f80d0320 0x7f11f81f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11f80d0ee0 0x7f11f8159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80ae5a0 con 0x7f11f815e190 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f80ae420 con 0x7f11f80d0ee0 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f816bad0 con 0x7f11f80d0320 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11f7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11f80d0320 0x7f11f81f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11f7400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11f80d0ee0 0x7f11f8159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42600/0 (socket says 172.21.15.28:42600) 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 -- 172.21.15.28:0/2973760490 learned_addr learned my addr 172.21.15.28:0/2973760490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 -- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11f80d0320 msgr2=0x7f11f81f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11f80d0320 0x7f11f81f6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 -- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11f80d0ee0 msgr2=0x7f11f8159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11f80d0ee0 0x7f11f8159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 -- 172.21.15.28:0/2973760490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f81f70b0 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fcc00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c790 tx=0x7f11f000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0003b80 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0003d20 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f000aa50 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f8159f80 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.888+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f815ade0 con 0x7f11f815e190 2024-03-21T08:42:10.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f80cfea0 con 0x7f11f815e190 2024-03-21T08:42:10.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f11f000abf0 con 0x7f11f815e190 2024-03-21T08:42:10.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11efe00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 0x7f11d806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:10.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f11f0086130 con 0x7f11f815e190 2024-03-21T08:42:10.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11f7e00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 0x7f11d806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.892+0000 7f11f7e00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 0x7f11d806a030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f11e8007fd0 tx=0x7f11e8007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:10.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:10.896+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f11f0056150 con 0x7f11f815e190 2024-03-21T08:42:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11f804f4f0 con 0x7f11f815e190 2024-03-21T08:42:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11efe00640 1 -- 172.21.15.28:0/2973760490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11f005a1f0 con 0x7f11f815e190 2024-03-21T08:42:11.061 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 msgr2=0x7f11d806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 0x7f11d806a030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f11e8007fd0 tx=0x7f11e8007d80 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 msgr2=0x7f11f81528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c790 tx=0x7f11f000cc50 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 shutdown_connections 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f815e190 0x7f11f81528c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f11f80d0ee0 0x7f11f8159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f11d8067b70 0x7f11d806a030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 --2- 172.21.15.28:0/2973760490 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f11f80d0320 0x7f11f81f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 >> 172.21.15.28:0/2973760490 conn(0x7f11f8099a30 msgr2=0x7f11f806feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 shutdown_connections 2024-03-21T08:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.060+0000 7f11fec00640 1 -- 172.21.15.28:0/2973760490 wait complete. 2024-03-21T08:42:11.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 Processor -- start 2024-03-21T08:42:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 -- start start 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ac0071a80 0x7f7ac006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7ac015cbb0 0x7f7ac00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00ce5b0 con 0x7f7ac00cde40 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac015c720 con 0x7f7ac0071a80 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7ac5600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00ce890 con 0x7f7ac015cbb0 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7abea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ac0071a80 0x7f7ac006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7abe000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ac00cde40 0x7f7ac0071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39232/0 (socket says 172.21.15.28:39232) 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7abe000640 1 -- 172.21.15.28:0/40146891 learned_addr learned my addr 172.21.15.28:0/40146891 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7abea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ac0071a80 0x7f7ac006f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42614/0 (socket says 172.21.15.28:42614) 2024-03-21T08:42:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.224+0000 7f7abf400640 1 --2- 172.21.15.28:0/40146891 >> v2:172.21.15.151:3300/0 conn(0x7f7ac015cbb0 0x7f7ac00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abea00640 1 -- 172.21.15.28:0/40146891 >> v2:172.21.15.151:3300/0 conn(0x7f7ac015cbb0 msgr2=0x7f7ac00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abea00640 1 --2- 172.21.15.28:0/40146891 >> v2:172.21.15.151:3300/0 conn(0x7f7ac015cbb0 0x7f7ac00cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abea00640 1 -- 172.21.15.28:0/40146891 >> v1:172.21.15.28:6789/0 conn(0x7f7ac00cde40 legacy=0x7f7ac0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abea00640 1 -- 172.21.15.28:0/40146891 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac015c3b0 con 0x7f7ac0071a80 2024-03-21T08:42:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abea00640 1 --2- 172.21.15.28:0/40146891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac0071a80 0x7f7ac006f0a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00b5030 tx=0x7f7ab001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7811383304be1b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abd600640 1 -- 172.21.15.28:0/40146891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00ab610 con 0x7f7ac0071a80 2024-03-21T08:42:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abd600640 1 -- 172.21.15.28:0/40146891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00b0070 con 0x7f7ac0071a80 2024-03-21T08:42:11.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7abd600640 1 -- 172.21.15.28:0/40146891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00ab9d0 con 0x7f7ac0071a80 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 -- 172.21.15.28:0/40146891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac0071a80 msgr2=0x7f7ac006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 --2- 172.21.15.28:0/40146891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac0071a80 0x7f7ac006f0a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00b5030 tx=0x7f7ab001c3f0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 -- 172.21.15.28:0/40146891 shutdown_connections 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 --2- 172.21.15.28:0/40146891 >> v2:172.21.15.151:3300/0 conn(0x7f7ac015cbb0 0x7f7ac00cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 --2- 172.21.15.28:0/40146891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac0071a80 0x7f7ac006f0a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.228+0000 7f7ac5600640 1 -- 172.21.15.28:0/40146891 >> 172.21.15.28:0/40146891 conn(0x7f7ac00674f0 msgr2=0x7f7ac006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- 172.21.15.28:0/40146891 shutdown_connections 2024-03-21T08:42:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- 172.21.15.28:0/40146891 wait complete. 2024-03-21T08:42:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 Processor -- start 2024-03-21T08:42:11.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- start start 2024-03-21T08:42:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 0x7f7ac0140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ac00cde40 0x7f7ac013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0168f20 con 0x7f7ac015cbb0 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00ce220 con 0x7f7ac0071a80 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7ac5600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac01690a0 con 0x7f7ac00cde40 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42620/0 (socket says 172.21.15.28:42620) 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abf400640 1 -- 172.21.15.28:0/1521998929 learned_addr learned my addr 172.21.15.28:0/1521998929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 0x7f7ac0140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abe000640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ac00cde40 0x7f7ac013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 -- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ac00cde40 msgr2=0x7f7ac013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ac00cde40 0x7f7ac013a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 -- 172.21.15.28:0/1521998929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 msgr2=0x7f7ac013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 -- 172.21.15.28:0/1521998929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac013f400 con 0x7f7ac0071a80 2024-03-21T08:42:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abf400640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.232+0000 7f7abea00640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 0x7f7ac0140580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00ac000 tx=0x7f7ab001ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00bf610 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00bf7d0 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00bfbc0 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac013f6e0 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac013fca0 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac0073430 con 0x7f7ac0071a80 2024-03-21T08:42:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7ab00bd020 con 0x7f7ac0071a80 2024-03-21T08:42:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 0x7f7aa0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f7ab0120630 con 0x7f7ac0071a80 2024-03-21T08:42:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7abe000640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 0x7f7aa0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.236+0000 7f7abe000640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 0x7f7aa0069fe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00b2860 tx=0x7f7aac007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.240+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ab00c6070 con 0x7f7ac0071a80 2024-03-21T08:42:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.404+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ac004f4f0 con 0x7f7ac0071a80 2024-03-21T08:42:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.404+0000 7f7ab6a00640 1 -- 172.21.15.28:0/1521998929 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ab00a8050 con 0x7f7ac0071a80 2024-03-21T08:42:11.407 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 msgr2=0x7f7aa0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 0x7f7aa0069fe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00b2860 tx=0x7f7aac007e90 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 msgr2=0x7f7ac0140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 0x7f7ac0140580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00ac000 tx=0x7f7ab001ac30 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 shutdown_connections 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac015cbb0 0x7f7ac013a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7aa0067b20 0x7f7aa0069fe0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ac00cde40 0x7f7ac013a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 --2- 172.21.15.28:0/1521998929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ac0071a80 0x7f7ac0140580 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 >> 172.21.15.28:0/1521998929 conn(0x7f7ac00674f0 msgr2=0x7f7ac00b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 shutdown_connections 2024-03-21T08:42:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.408+0000 7f7ac5600640 1 -- 172.21.15.28:0/1521998929 wait complete. 2024-03-21T08:42:11.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 Processor -- start 2024-03-21T08:42:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 -- start start 2024-03-21T08:42:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0bc80b4420 0x7f0bc8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc80b4800 0x7f0bc8071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc815bf90 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc815b1d0 con 0x7f0bc80b4420 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcfc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc815b4b0 con 0x7f0bc80b4800 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42628/0 (socket says 172.21.15.28:42628) 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 -- 172.21.15.28:0/4124251578 learned_addr learned my addr 172.21.15.28:0/4124251578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bbfe00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc80b4800 0x7f0bc8071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcd200640 1 --2- 172.21.15.28:0/4124251578 >> v2:172.21.15.151:3300/0 conn(0x7f0bc80b4420 0x7f0bc8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 -- 172.21.15.28:0/4124251578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc80b4800 msgr2=0x7f0bc8071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc80b4800 0x7f0bc8071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 -- 172.21.15.28:0/4124251578 >> v2:172.21.15.151:3300/0 conn(0x7f0bc80b4420 msgr2=0x7f0bc8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 --2- 172.21.15.28:0/4124251578 >> v2:172.21.15.151:3300/0 conn(0x7f0bc80b4420 0x7f0bc8071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.572+0000 7f0bcdc00640 1 -- 172.21.15.28:0/4124251578 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc815b790 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcdc00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0bc806ecd0 tx=0x7f0bc001ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa9e60657d5a9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bc7e00640 1 -- 172.21.15.28:0/4124251578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc000b2d0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bc7e00640 1 -- 172.21.15.28:0/4124251578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc000b470 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bc7e00640 1 -- 172.21.15.28:0/4124251578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc00a7c90 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 -- 172.21.15.28:0/4124251578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 msgr2=0x7f0bc80cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0bc806ecd0 tx=0x7f0bc001ebc0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 -- 172.21.15.28:0/4124251578 shutdown_connections 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc80cd0a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/4124251578 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc80b4800 0x7f0bc8071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/4124251578 >> v2:172.21.15.151:3300/0 conn(0x7f0bc80b4420 0x7f0bc8071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 -- 172.21.15.28:0/4124251578 >> 172.21.15.28:0/4124251578 conn(0x7f0bc80674f0 msgr2=0x7f0bc806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 -- 172.21.15.28:0/4124251578 shutdown_connections 2024-03-21T08:42:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.576+0000 7f0bcfc00640 1 -- 172.21.15.28:0/4124251578 wait complete. 2024-03-21T08:42:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 Processor -- start 2024-03-21T08:42:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- start start 2024-03-21T08:42:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bc80b4420 0x7f0bc8146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 0x7f0bc815fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc815efb0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc815ee30 con 0x7f0bc80b4420 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc815f130 con 0x7f0bc8163810 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42638/0 (socket says 172.21.15.28:42638) 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 -- 172.21.15.28:0/573388729 learned_addr learned my addr 172.21.15.28:0/573388729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcd200640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bc80b4420 0x7f0bc8146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcdc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 0x7f0bc815fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 -- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 msgr2=0x7f0bc815fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 0x7f0bc815fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 -- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bc80b4420 msgr2=0x7f0bc8146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bc80b4420 0x7f0bc8146a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 -- 172.21.15.28:0/573388729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8160a00 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcdc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 0x7f0bc815fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bbfe00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000e930 tx=0x7f0bb000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0017070 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000ca00 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000cdd0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc8160ce0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc8133440 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.580+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc8073430 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.584+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0bb001a070 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.584+0000 7f0bc5600640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 0x7f0ba406a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.584+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f0bb0086550 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.584+0000 7f0bcd200640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 0x7f0ba406a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.584+0000 7f0bcd200640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 0x7f0ba406a060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009b60 tx=0x7f0bb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.588+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0bb00565a0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.748+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0bc804f4f0 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bc5600640 1 -- 172.21.15.28:0/573388729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0bb005a640 con 0x7f0bc80ccbf0 2024-03-21T08:42:11.752 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 msgr2=0x7f0ba406a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 0x7f0ba406a060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009b60 tx=0x7f0bb8009190 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 msgr2=0x7f0bc81632d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000e930 tx=0x7f0bb000edf0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 shutdown_connections 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0ba4067ba0 0x7f0ba406a060 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0bc8163810 0x7f0bc815fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bc80ccbf0 0x7f0bc81632d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 --2- 172.21.15.28:0/573388729 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0bc80b4420 0x7f0bc8146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 >> 172.21.15.28:0/573388729 conn(0x7f0bc80674f0 msgr2=0x7f0bc814a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 shutdown_connections 2024-03-21T08:42:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.752+0000 7f0bcfc00640 1 -- 172.21.15.28:0/573388729 wait complete. 2024-03-21T08:42:11.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.912+0000 7fc7b0000640 1 Processor -- start 2024-03-21T08:42:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.912+0000 7fc7b0000640 1 -- start start 2024-03-21T08:42:11.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7b0000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7a8144690 0x7fc7a80b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7b0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8071a80 0x7fc7a8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7b0000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a80723e0 con 0x7fc7a81442b0 2024-03-21T08:42:11.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7b0000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a80b2ad0 con 0x7fc7a8144690 2024-03-21T08:42:11.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7b0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80b2d20 con 0x7fc7a8071a80 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7a8144690 0x7fc7a80b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7a8144690 0x7fc7a80b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42640/0 (socket says 172.21.15.28:42640) 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 -- 172.21.15.28:0/2405109815 learned_addr learned my addr 172.21.15.28:0/2405109815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7ad600640 1 --1- 172.21.15.28:0/2405109815 >> v1:172.21.15.28:6789/0 conn(0x7fc7a81442b0 0x7fc7a80aebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39248/0 (socket says 172.21.15.28:39248) 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7ae000640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8071a80 0x7fc7a8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 -- 172.21.15.28:0/2405109815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8071a80 msgr2=0x7fc7a8071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8071a80 0x7fc7a8071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 -- 172.21.15.28:0/2405109815 >> v1:172.21.15.28:6789/0 conn(0x7fc7a81442b0 legacy=0x7fc7a80aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.916+0000 7fc7acc00640 1 -- 172.21.15.28:0/2405109815 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80b3000 con 0x7fc7a8144690 2024-03-21T08:42:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7acc00640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8144690 0x7fc7a80b3dc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc794009870 tx=0x7fc79401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d75d5073043f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7a7e00640 1 -- 172.21.15.28:0/2405109815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940a95f0 con 0x7fc7a8144690 2024-03-21T08:42:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7a7e00640 1 -- 172.21.15.28:0/2405109815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7940a9790 con 0x7fc7a8144690 2024-03-21T08:42:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7a7e00640 1 -- 172.21.15.28:0/2405109815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7940a9b60 con 0x7fc7a8144690 2024-03-21T08:42:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- 172.21.15.28:0/2405109815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8144690 msgr2=0x7fc7a80b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8144690 0x7fc7a80b3dc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc794009870 tx=0x7fc79401c530 comp rx=0 tx=0).stop 2024-03-21T08:42:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- 172.21.15.28:0/2405109815 shutdown_connections 2024-03-21T08:42:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8071a80 0x7fc7a8071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 --2- 172.21.15.28:0/2405109815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8144690 0x7fc7a80b3dc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- 172.21.15.28:0/2405109815 >> 172.21.15.28:0/2405109815 conn(0x7fc7a80674f0 msgr2=0x7fc7a806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- 172.21.15.28:0/2405109815 shutdown_connections 2024-03-21T08:42:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- 172.21.15.28:0/2405109815 wait complete. 2024-03-21T08:42:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 Processor -- start 2024-03-21T08:42:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 -- start start 2024-03-21T08:42:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.920+0000 7fc7b0000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81442b0 0x7fc7a8144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8144690 0x7fc7a8144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80af470 con 0x7fc7a8071a80 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a80af2f0 con 0x7fc7a81442b0 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a806e2b0 con 0x7fc7a8144690 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42650/0 (socket says 172.21.15.28:42650) 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 -- 172.21.15.28:0/833813156 learned_addr learned my addr 172.21.15.28:0/833813156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7acc00640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81442b0 0x7fc7a8144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ae000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8144690 0x7fc7a8144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 -- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8144690 msgr2=0x7fc7a8144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8144690 0x7fc7a8144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 -- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81442b0 msgr2=0x7fc7a8144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81442b0 0x7fc7a8144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 -- 172.21.15.28:0/833813156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80af6e0 con 0x7fc7a8071a80 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7ad600640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c420 tx=0x7fc79c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0074e0 con 0x7fc7a8071a80 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c007680 con 0x7fc7a8071a80 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c004a30 con 0x7fc7a8071a80 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a80af950 con 0x7fc7a8071a80 2024-03-21T08:42:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a80aff10 con 0x7fc7a8071a80 2024-03-21T08:42:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8073430 con 0x7fc7a8071a80 2024-03-21T08:42:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.924+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc79c01a020 con 0x7fc7a8071a80 2024-03-21T08:42:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.928+0000 7fc7a5600640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 0x7fc788069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.928+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fc79c085a50 con 0x7fc7a8071a80 2024-03-21T08:42:11.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.928+0000 7fc7acc00640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 0x7fc788069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:11.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.928+0000 7fc7acc00640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 0x7fc788069fe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7940ae000 tx=0x7fc794009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:11.932+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc79c055a00 con 0x7fc7a8071a80 2024-03-21T08:42:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.108+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7a804f4f0 con 0x7fc7a8071a80 2024-03-21T08:42:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.108+0000 7fc7a5600640 1 -- 172.21.15.28:0/833813156 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc79c059aa0 con 0x7fc7a8071a80 2024-03-21T08:42:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 msgr2=0x7fc788069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 0x7fc788069fe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7940ae000 tx=0x7fc794009040 comp rx=0 tx=0).stop 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 msgr2=0x7fc7a813b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c420 tx=0x7fc79c00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 shutdown_connections 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc788067b20 0x7fc788069fe0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc7a8144690 0x7fc7a8144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc7a81442b0 0x7fc7a8144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 --2- 172.21.15.28:0/833813156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7a8071a80 0x7fc7a813b560 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 >> 172.21.15.28:0/833813156 conn(0x7fc7a80674f0 msgr2=0x7fc7a806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 shutdown_connections 2024-03-21T08:42:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.112+0000 7fc7b0000640 1 -- 172.21.15.28:0/833813156 wait complete. 2024-03-21T08:42:12.133 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:42:12.133 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:12.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 Processor -- start 2024-03-21T08:42:12.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 -- start start 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 0x7f4b1815f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180cfac0 con 0x7f4b180d0880 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180cfda0 con 0x7f4b18099b80 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1f000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180d0080 con 0x7f4b180b0b40 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b0f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4b180d0880 0x7f4b1806c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39252/0 (socket says 172.21.15.28:39252) 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b0f400640 1 -- 172.21.15.28:0/4108934944 learned_addr learned my addr 172.21.15.28:0/4108934944 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1d000640 1 --2- 172.21.15.28:0/4108934944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 0x7f4b1815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524073874 0 0) 0x7f4b180cfac0 con 0x7f4b180d0880 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc003530 con 0x7f4b180d0880 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579341399 0 0) 0x7f4b180cfda0 con 0x7f4b18099b80 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b180cfac0 con 0x7f4b18099b80 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1d000640 1 -- 172.21.15.28:0/4108934944 >> v1:172.21.15.151:6789/0 conn(0x7f4b18099b80 legacy=0x7f4b18099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1d000640 1 -- 172.21.15.28:0/4108934944 >> v1:172.21.15.28:6789/0 conn(0x7f4b180d0880 legacy=0x7f4b1806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.284+0000 7f4b1d000640 1 -- 172.21.15.28:0/4108934944 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b180d0360 con 0x7f4b180b0b40 2024-03-21T08:42:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1d000640 1 --2- 172.21.15.28:0/4108934944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 0x7f4b1815f220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097f0 tx=0x7f4b1001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ea2449a64385a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100ab600 con 0x7f4b180b0b40 2024-03-21T08:42:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b100b0070 con 0x7f4b180b0b40 2024-03-21T08:42:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b17400640 1 -- 172.21.15.28:0/4108934944 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100ab9c0 con 0x7f4b180b0b40 2024-03-21T08:42:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 -- 172.21.15.28:0/4108934944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 msgr2=0x7f4b1815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4108934944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 0x7f4b1815f220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097f0 tx=0x7f4b1001c4e0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 -- 172.21.15.28:0/4108934944 shutdown_connections 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4108934944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b180b0b40 0x7f4b1815f220 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 -- 172.21.15.28:0/4108934944 >> 172.21.15.28:0/4108934944 conn(0x7f4b180b3510 msgr2=0x7f4b180b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 -- 172.21.15.28:0/4108934944 shutdown_connections 2024-03-21T08:42:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.288+0000 7f4b1f000640 1 -- 172.21.15.28:0/4108934944 wait complete. 2024-03-21T08:42:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 Processor -- start 2024-03-21T08:42:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 -- start start 2024-03-21T08:42:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b18099b80 0x7f4b1815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b180b0b40 0x7f4b1815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1816c3f0 con 0x7f4b180d0880 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1816c270 con 0x7f4b180b0b40 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1816c570 con 0x7f4b18099b80 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b0f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b180b0b40 0x7f4b1815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42652/0 (socket says 172.21.15.28:42652) 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 -- 172.21.15.28:0/4188266474 learned_addr learned my addr 172.21.15.28:0/4188266474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b17e00640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b18099b80 0x7f4b1815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 -- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b18099b80 msgr2=0x7f4b1815bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b18099b80 0x7f4b1815bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 -- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b180b0b40 msgr2=0x7f4b1815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b180b0b40 0x7f4b1815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 -- 172.21.15.28:0/4188266474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b181f8a40 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1d000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097f0 tx=0x7f4b1001ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100bf410 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b100bf5b0 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100bf980 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.292+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1815c660 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1815cc10 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b181f8c10 con 0x7f4b180d0880 2024-03-21T08:42:12.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4b100bd020 con 0x7f4b180d0880 2024-03-21T08:42:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b14c00640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 0x7f4af4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f4b10120240 con 0x7f4b180d0880 2024-03-21T08:42:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b17e00640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 0x7f4af4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.296+0000 7f4b17e00640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 0x7f4af4069fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007fd0 tx=0x7f4b08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.300+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4b100c6030 con 0x7f4b180d0880 2024-03-21T08:42:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.464+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b180b43f0 con 0x7f4b180d0880 2024-03-21T08:42:12.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.464+0000 7f4b14c00640 1 -- 172.21.15.28:0/4188266474 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f4b100160d0 con 0x7f4b180d0880 2024-03-21T08:42:12.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:12.467 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 msgr2=0x7f4af4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 0x7f4af4069fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007fd0 tx=0x7f4b08009040 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 msgr2=0x7f4b181f7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097f0 tx=0x7f4b1001ad20 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 shutdown_connections 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b180d0880 0x7f4b181f7580 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b180b0b40 0x7f4b1815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4af4067b20 0x7f4af4069fe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 --2- 172.21.15.28:0/4188266474 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b18099b80 0x7f4b1815bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 >> 172.21.15.28:0/4188266474 conn(0x7f4b180b3510 msgr2=0x7f4b18099f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 shutdown_connections 2024-03-21T08:42:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.468+0000 7f4b1f000640 1 -- 172.21.15.28:0/4188266474 wait complete. 2024-03-21T08:42:12.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:12.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 Processor -- start 2024-03-21T08:42:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 -- start start 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1e5c1580c0 0x7f1e5c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0701b0 con 0x7f1e5c0b48d0 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c070490 con 0x7f1e5c06b6b0 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e60a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5c070770 con 0x7f1e5c1580c0 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5b200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1e5c0b48d0 0x7f1e5c0d0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39268/0 (socket says 172.21.15.28:39268) 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5b200640 1 -- 172.21.15.28:0/3521755588 learned_addr learned my addr 172.21.15.28:0/3521755588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5a800640 1 --2- 172.21.15.28:0/3521755588 >> v2:172.21.15.151:3300/0 conn(0x7f1e5c1580c0 0x7f1e5c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606659396 0 0) 0x7f1e5c0701b0 con 0x7f1e5c0b48d0 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e40003530 con 0x7f1e5c0b48d0 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5a800640 1 -- 172.21.15.28:0/3521755588 >> v1:172.21.15.151:6789/0 conn(0x7f1e5c06b6b0 legacy=0x7f1e5c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5a800640 1 -- 172.21.15.28:0/3521755588 >> v1:172.21.15.28:6789/0 conn(0x7f1e5c0b48d0 legacy=0x7f1e5c0d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000707560 0 0) 0x7f1e5c070490 con 0x7f1e5c06b6b0 2024-03-21T08:42:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.632+0000 7f1e5a800640 1 -- 172.21.15.28:0/3521755588 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c070a50 con 0x7f1e5c1580c0 2024-03-21T08:42:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e5a800640 1 --2- 172.21.15.28:0/3521755588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c06b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097c0 tx=0x7f1e4c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9849a03db4e46d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0b0070 con 0x7f1e5c1580c0 2024-03-21T08:42:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4c0ab600 con 0x7f1e5c1580c0 2024-03-21T08:42:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e59e00640 1 -- 172.21.15.28:0/3521755588 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0ab9d0 con 0x7f1e5c1580c0 2024-03-21T08:42:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 -- 172.21.15.28:0/3521755588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c1580c0 msgr2=0x7f1e5c06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3521755588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c06b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097c0 tx=0x7f1e4c01c410 comp rx=0 tx=0).stop 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 -- 172.21.15.28:0/3521755588 shutdown_connections 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3521755588 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c06b170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 -- 172.21.15.28:0/3521755588 >> 172.21.15.28:0/3521755588 conn(0x7f1e5c0b23d0 msgr2=0x7f1e5c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 -- 172.21.15.28:0/3521755588 shutdown_connections 2024-03-21T08:42:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.636+0000 7f1e60a00640 1 -- 172.21.15.28:0/3521755588 wait complete. 2024-03-21T08:42:12.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 Processor -- start 2024-03-21T08:42:12.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- start start 2024-03-21T08:42:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c06b6b0 0x7f1e5c14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e5c0b48d0 0x7f1e5c14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c16c290 con 0x7f1e5c1580c0 2024-03-21T08:42:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0d15d0 con 0x7f1e5c06b6b0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c16c410 con 0x7f1e5c0b48d0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42656/0 (socket says 172.21.15.28:42656) 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5a800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c06b6b0 0x7f1e5c14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 -- 172.21.15.28:0/3116514491 learned_addr learned my addr 172.21.15.28:0/3116514491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e53e00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e5c0b48d0 0x7f1e5c14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 -- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e5c0b48d0 msgr2=0x7f1e5c14e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e5c0b48d0 0x7f1e5c14e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 -- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c06b6b0 msgr2=0x7f1e5c14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c06b6b0 0x7f1e5c14e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 -- 172.21.15.28:0/3116514491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c0d06c0 con 0x7f1e5c1580c0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e5b200640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400cb60 tx=0x7f1e5400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54017070 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5400ad40 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c0d0960 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54012600 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.640+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c1f8410 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c04f410 con 0x7f1e5c1580c0 2024-03-21T08:42:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1e5401a070 con 0x7f1e5c1580c0 2024-03-21T08:42:12.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e52a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 0x7f1e38069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f1e54085fa0 con 0x7f1e5c1580c0 2024-03-21T08:42:12.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e5a800640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 0x7f1e38069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.644+0000 7f1e5a800640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 0x7f1e38069fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0ad000 tx=0x7f1e4c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.648+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1e54055f40 con 0x7f1e5c1580c0 2024-03-21T08:42:12.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e5c14ff40 con 0x7f1e5c1580c0 2024-03-21T08:42:12.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e52a00640 1 -- 172.21.15.28:0/3116514491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e54059fe0 con 0x7f1e5c1580c0 2024-03-21T08:42:12.818 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 msgr2=0x7f1e38069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 0x7f1e38069fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0ad000 tx=0x7f1e4c0a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 msgr2=0x7f1e5c1486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400cb60 tx=0x7f1e5400a430 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 shutdown_connections 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e5c1580c0 0x7f1e5c1486b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1e38067b20 0x7f1e38069fe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1e5c0b48d0 0x7f1e5c14e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 --2- 172.21.15.28:0/3116514491 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1e5c06b6b0 0x7f1e5c14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 >> 172.21.15.28:0/3116514491 conn(0x7f1e5c0b23d0 msgr2=0x7f1e5c0c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 shutdown_connections 2024-03-21T08:42:12.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.816+0000 7f1e60a00640 1 -- 172.21.15.28:0/3116514491 wait complete. 2024-03-21T08:42:12.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.984+0000 7f84f1a00640 1 Processor -- start 2024-03-21T08:42:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.984+0000 7f84f1a00640 1 -- start start 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84f1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 0x7f84ec0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84f1a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 0x7f84ec0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84f1a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0cd670 con 0x7f84ec0b4420 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84f1a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec15bf90 con 0x7f84ec072190 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84f1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec15b1d0 con 0x7f84ec0b4800 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84eb400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 0x7f84ec0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84eaa00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f84ec0b4420 0x7f84ec071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42440/0 (socket says 172.21.15.28:42440) 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84eaa00640 1 -- 172.21.15.28:0/1745679651 learned_addr learned my addr 172.21.15.28:0/1745679651 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84e2000640 1 --2- 172.21.15.28:0/1745679651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 0x7f84ec0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84e2000640 1 -- 172.21.15.28:0/1745679651 >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 msgr2=0x7f84ec0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84e2000640 1 --2- 172.21.15.28:0/1745679651 >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 0x7f84ec0cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84e2000640 1 -- 172.21.15.28:0/1745679651 >> v1:172.21.15.151:6789/0 conn(0x7f84ec0b4420 legacy=0x7f84ec071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84e2000640 1 -- 172.21.15.28:0/1745679651 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec15b4b0 con 0x7f84ec0b4800 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84ea000640 1 -- 172.21.15.28:0/1745679651 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411644 0 0) 0x7f84ec0cd670 con 0x7f84ec0b4420 2024-03-21T08:42:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.988+0000 7f84eb400640 1 --2- 172.21.15.28:0/1745679651 >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 0x7f84ec0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84e2000640 1 --2- 172.21.15.28:0/1745679651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 0x7f84ec0ccbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84d40097c0 tx=0x7f84d401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49816938f841e779 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84ea000640 1 -- 172.21.15.28:0/1745679651 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b0070 con 0x7f84ec0b4800 2024-03-21T08:42:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84ea000640 1 -- 172.21.15.28:0/1745679651 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d40ab600 con 0x7f84ec0b4800 2024-03-21T08:42:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84ea000640 1 -- 172.21.15.28:0/1745679651 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40ab9d0 con 0x7f84ec0b4800 2024-03-21T08:42:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- 172.21.15.28:0/1745679651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 msgr2=0x7f84ec0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- 172.21.15.28:0/1745679651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 0x7f84ec0ccbf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84d40097c0 tx=0x7f84d401c470 comp rx=0 tx=0).stop 2024-03-21T08:42:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- 172.21.15.28:0/1745679651 shutdown_connections 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- 172.21.15.28:0/1745679651 >> v2:172.21.15.151:3300/0 conn(0x7f84ec072190 0x7f84ec0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- 172.21.15.28:0/1745679651 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4800 0x7f84ec0ccbf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- 172.21.15.28:0/1745679651 >> 172.21.15.28:0/1745679651 conn(0x7f84ec0674f0 msgr2=0x7f84ec06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- 172.21.15.28:0/1745679651 shutdown_connections 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- 172.21.15.28:0/1745679651 wait complete. 2024-03-21T08:42:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 Processor -- start 2024-03-21T08:42:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- start start 2024-03-21T08:42:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 0x7f84ec14f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84ec148db0 0x7f84ec157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec06e430 con 0x7f84ec072190 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec06e2b0 con 0x7f84ec148db0 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.992+0000 7f84f1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec06e5b0 con 0x7f84ec0b4420 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42666/0 (socket says 172.21.15.28:42666) 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 -- 172.21.15.28:0/158053159 learned_addr learned my addr 172.21.15.28:0/158053159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e2000640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 0x7f84ec14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eb400640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84ec148db0 0x7f84ec157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 -- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 msgr2=0x7f84ec14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 0x7f84ec14f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 -- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84ec148db0 msgr2=0x7f84ec157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84ec148db0 0x7f84ec157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 -- 172.21.15.28:0/158053159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec14fea0 con 0x7f84ec072190 2024-03-21T08:42:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e2000640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 0x7f84ec14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84eaa00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c520 tx=0x7f84dc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc018070 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc0074e0 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0049f0 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec151fb0 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec152570 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec073430 con 0x7f84ec072190 2024-03-21T08:42:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:12.996+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f84dc004b90 con 0x7f84ec072190 2024-03-21T08:42:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.000+0000 7f84e3400640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 0x7f84c8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.000+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f84dc085630 con 0x7f84ec072190 2024-03-21T08:42:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.000+0000 7f84e2000640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 0x7f84c8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.000+0000 7f84e2000640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 0x7f84c8069fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f84d40ad000 tx=0x7f84d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.004+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f84dc0555c0 con 0x7f84ec072190 2024-03-21T08:42:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.164+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84ec1500d0 con 0x7f84ec072190 2024-03-21T08:42:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.164+0000 7f84e3400640 1 -- 172.21.15.28:0/158053159 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84dc059660 con 0x7f84ec072190 2024-03-21T08:42:13.168 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 msgr2=0x7f84c8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 0x7f84c8069fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f84d40ad000 tx=0x7f84d40a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 msgr2=0x7f84ec153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c520 tx=0x7f84dc00c9e0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 shutdown_connections 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f84ec148db0 0x7f84ec157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f84c8067b20 0x7f84c8069fe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f84ec0b4420 0x7f84ec14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 --2- 172.21.15.28:0/158053159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec072190 0x7f84ec153cb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 >> 172.21.15.28:0/158053159 conn(0x7f84ec0674f0 msgr2=0x7f84ec15c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 shutdown_connections 2024-03-21T08:42:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.168+0000 7f84f1a00640 1 -- 172.21.15.28:0/158053159 wait complete. 2024-03-21T08:42:13.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 Processor -- start 2024-03-21T08:42:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 -- start start 2024-03-21T08:42:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd63009a3a0 0x7fd63006cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6300b1110 0x7fd63006d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6300cc3d0 con 0x7fd63015dff0 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6300cc180 con 0x7fd63009a3a0 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd636c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300cc6b0 con 0x7fd6300b1110 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd62fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd63015dff0 0x7fd630099c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39272/0 (socket says 172.21.15.28:39272) 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd62fe00640 1 -- 172.21.15.28:0/1881773166 learned_addr learned my addr 172.21.15.28:0/1881773166 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd634c00640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6300b1110 0x7fd63006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd627400640 1 --2- 172.21.15.28:0/1881773166 >> v2:172.21.15.151:3300/0 conn(0x7fd63009a3a0 0x7fd63006cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd62f400640 1 -- 172.21.15.28:0/1881773166 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703324063 0 0) 0x7fd6300cc3d0 con 0x7fd63015dff0 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd62f400640 1 -- 172.21.15.28:0/1881773166 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd614003530 con 0x7fd63015dff0 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd627400640 1 -- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6300b1110 msgr2=0x7fd63006d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd627400640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6300b1110 0x7fd63006d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd627400640 1 -- 172.21.15.28:0/1881773166 >> v1:172.21.15.28:6789/0 conn(0x7fd63015dff0 legacy=0x7fd630099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.332+0000 7fd627400640 1 -- 172.21.15.28:0/1881773166 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300cc990 con 0x7fd63009a3a0 2024-03-21T08:42:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd627400640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd63009a3a0 0x7fd63006cc00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd6180097c0 tx=0x7fd61801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=876e5f2da09e221d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd62f400640 1 -- 172.21.15.28:0/1881773166 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180b0070 con 0x7fd63009a3a0 2024-03-21T08:42:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd62f400640 1 -- 172.21.15.28:0/1881773166 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180ab600 con 0x7fd63009a3a0 2024-03-21T08:42:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd62f400640 1 -- 172.21.15.28:0/1881773166 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180ab9d0 con 0x7fd63009a3a0 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 -- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd63009a3a0 msgr2=0x7fd63006cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd63009a3a0 0x7fd63006cc00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd6180097c0 tx=0x7fd61801c380 comp rx=0 tx=0).stop 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 -- 172.21.15.28:0/1881773166 shutdown_connections 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6300b1110 0x7fd63006d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 --2- 172.21.15.28:0/1881773166 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd63009a3a0 0x7fd63006cc00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 -- 172.21.15.28:0/1881773166 >> 172.21.15.28:0/1881773166 conn(0x7fd6300b3620 msgr2=0x7fd6300b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:13.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 -- 172.21.15.28:0/1881773166 shutdown_connections 2024-03-21T08:42:13.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.336+0000 7fd636c00640 1 -- 172.21.15.28:0/1881773166 wait complete. 2024-03-21T08:42:13.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 Processor -- start 2024-03-21T08:42:13.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- start start 2024-03-21T08:42:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd63009a3a0 0x7fd6300ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 0x7fd6300ce600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300d0a60 con 0x7fd63015dff0 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300d08e0 con 0x7fd6300b1110 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300d0be0 con 0x7fd63009a3a0 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42674/0 (socket says 172.21.15.28:42674) 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 -- 172.21.15.28:0/127384261 learned_addr learned my addr 172.21.15.28:0/127384261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd627400640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 0x7fd6300ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd62fe00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd63009a3a0 0x7fd6300ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 -- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd63009a3a0 msgr2=0x7fd6300ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd63009a3a0 0x7fd6300ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 -- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 msgr2=0x7fd6300ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 0x7fd6300ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 -- 172.21.15.28:0/127384261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63015fcc0 con 0x7fd63015dff0 2024-03-21T08:42:13.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd627400640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 0x7fd6300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:13.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd634c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd62800efc0 tx=0x7fd62800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628018020 con 0x7fd63015dff0 2024-03-21T08:42:13.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628014480 con 0x7fd63015dff0 2024-03-21T08:42:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628014870 con 0x7fd63015dff0 2024-03-21T08:42:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63015b4f0 con 0x7fd63015dff0 2024-03-21T08:42:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.340+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63015bab0 con 0x7fd63015dff0 2024-03-21T08:42:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63009faf0 con 0x7fd63015dff0 2024-03-21T08:42:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd62801b020 con 0x7fd63015dff0 2024-03-21T08:42:13.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd62cc00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 0x7fd60c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fd628086ff0 con 0x7fd63015dff0 2024-03-21T08:42:13.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd62fe00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 0x7fd60c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.344+0000 7fd62fe00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 0x7fd60c069fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd620007fd0 tx=0x7fd620009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.348+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd628022020 con 0x7fd63015dff0 2024-03-21T08:42:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd630071b00 con 0x7fd63015dff0 2024-03-21T08:42:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd62cc00640 1 -- 172.21.15.28:0/127384261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd628010040 con 0x7fd63015dff0 2024-03-21T08:42:13.513 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:13.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 msgr2=0x7fd60c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 0x7fd60c069fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd620007fd0 tx=0x7fd620009040 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 msgr2=0x7fd63015ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd62800efc0 tx=0x7fd62800c490 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 shutdown_connections 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd63015dff0 0x7fd63015ec80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6300b1110 0x7fd6300ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd60c067b20 0x7fd60c069fe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 --2- 172.21.15.28:0/127384261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd63009a3a0 0x7fd6300ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 >> 172.21.15.28:0/127384261 conn(0x7fd6300b3620 msgr2=0x7fd6300b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 shutdown_connections 2024-03-21T08:42:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.512+0000 7fd636c00640 1 -- 172.21.15.28:0/127384261 wait complete. 2024-03-21T08:42:13.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.676+0000 7fd6bae00640 1 Processor -- start 2024-03-21T08:42:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.676+0000 7fd6bae00640 1 -- start start 2024-03-21T08:42:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6bae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 0x7fd6b4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6bae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6b40cd300 0x7fd6b4071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6bae00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415bf90 con 0x7fd6b40b4800 2024-03-21T08:42:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6bae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b415b1d0 con 0x7fd6b40cd300 2024-03-21T08:42:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6bae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b415b4b0 con 0x7fd6b40b4420 2024-03-21T08:42:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6b40cd300 0x7fd6b4071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 0x7fd6b4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6ab400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd6b40b4800 0x7fd6b40ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42454/0 (socket says 172.21.15.28:42454) 2024-03-21T08:42:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6ab400640 1 -- 172.21.15.28:0/2030289737 learned_addr learned my addr 172.21.15.28:0/2030289737 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b3400640 1 -- 172.21.15.28:0/2030289737 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007308535 0 0) 0x7fd6b415bf90 con 0x7fd6b40b4800 2024-03-21T08:42:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b3400640 1 -- 172.21.15.28:0/2030289737 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd698003530 con 0x7fd6b40b4800 2024-03-21T08:42:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b8e00640 1 -- 172.21.15.28:0/2030289737 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 msgr2=0x7fd6b4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b8e00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 0x7fd6b4071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b8e00640 1 -- 172.21.15.28:0/2030289737 >> v1:172.21.15.151:6789/0 conn(0x7fd6b40b4800 legacy=0x7fd6b40ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b8e00640 1 -- 172.21.15.28:0/2030289737 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b415b790 con 0x7fd6b40cd300 2024-03-21T08:42:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.680+0000 7fd6b3e00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 0x7fd6b4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6b8e00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b4071fc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009d90 tx=0x7fd6ac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e70144186c4b67d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6b3400640 1 -- 172.21.15.28:0/2030289737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0b2070 con 0x7fd6b40cd300 2024-03-21T08:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6b3400640 1 -- 172.21.15.28:0/2030289737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac0a5800 con 0x7fd6b40cd300 2024-03-21T08:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6b3400640 1 -- 172.21.15.28:0/2030289737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0a5bd0 con 0x7fd6b40cd300 2024-03-21T08:42:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- 172.21.15.28:0/2030289737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 msgr2=0x7fd6b4071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b4071fc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009d90 tx=0x7fd6ac01c8b0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- 172.21.15.28:0/2030289737 shutdown_connections 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b4071fc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 --2- 172.21.15.28:0/2030289737 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b40b4420 0x7fd6b4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- 172.21.15.28:0/2030289737 >> 172.21.15.28:0/2030289737 conn(0x7fd6b40674f0 msgr2=0x7fd6b406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- 172.21.15.28:0/2030289737 shutdown_connections 2024-03-21T08:42:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- 172.21.15.28:0/2030289737 wait complete. 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 Processor -- start 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.684+0000 7fd6bae00640 1 -- start start 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 0x7fd6b4146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b4146e20 0x7fd6b4156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b406e680 con 0x7fd6b40cd300 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b406e500 con 0x7fd6b40b4420 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b406e800 con 0x7fd6b4146e20 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 0x7fd6b4146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42694/0 (socket says 172.21.15.28:42694) 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 0x7fd6b4146820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35616/0 (socket says 172.21.15.28:35616) 2024-03-21T08:42:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 -- 172.21.15.28:0/3405730873 learned_addr learned my addr 172.21.15.28:0/3405730873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b8e00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b4146e20 0x7fd6b4156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 -- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b4146e20 msgr2=0x7fd6b4156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b4146e20 0x7fd6b4156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 -- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 msgr2=0x7fd6b4146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 0x7fd6b4146820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 -- 172.21.15.28:0/3405730873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4157330 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6ab400640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c390 tx=0x7fd69c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c017070 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c005d10 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c007310 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b4154d70 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b414f1e0 con 0x7fd6b40cd300 2024-03-21T08:42:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b4073430 con 0x7fd6b40cd300 2024-03-21T08:42:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.688+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd69c01a070 con 0x7fd6b40cd300 2024-03-21T08:42:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.692+0000 7fd6b0c00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 0x7fd690069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.692+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fd69c0856f0 con 0x7fd6b40cd300 2024-03-21T08:42:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.692+0000 7fd6b3e00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 0x7fd690069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:13.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.692+0000 7fd6b3e00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 0x7fd690069fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400b820 tx=0x7fd6a400b3a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.696+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd69c0556c0 con 0x7fd6b40cd300 2024-03-21T08:42:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.856+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd6b404f4f0 con 0x7fd6b40cd300 2024-03-21T08:42:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.856+0000 7fd6b0c00640 1 -- 172.21.15.28:0/3405730873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd69c059760 con 0x7fd6b40cd300 2024-03-21T08:42:13.859 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 msgr2=0x7fd690069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 0x7fd690069fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400b820 tx=0x7fd6a400b3a0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 msgr2=0x7fd6b41530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c390 tx=0x7fd69c00c850 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 shutdown_connections 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd690067b20 0x7fd690069fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd6b4146e20 0x7fd6b4156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6b40cd300 0x7fd6b41530e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 --2- 172.21.15.28:0/3405730873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd6b40b4420 0x7fd6b4146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 >> 172.21.15.28:0/3405730873 conn(0x7fd6b40674f0 msgr2=0x7fd6b4151310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 shutdown_connections 2024-03-21T08:42:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:13.860+0000 7fd6bae00640 1 -- 172.21.15.28:0/3405730873 wait complete. 2024-03-21T08:42:13.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.020+0000 7f32e2800640 1 Processor -- start 2024-03-21T08:42:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.020+0000 7f32e2800640 1 -- start start 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e2800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f32dc15dba0 0x7f32dc0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32dc0ce3d0 con 0x7f32dc071c90 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e2800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32dc0ce6b0 con 0x7f32dc0d0140 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e2800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32dc0cc340 con 0x7f32dc15dba0 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e0a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32dc071c90 0x7f32dc0cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39278/0 (socket says 172.21.15.28:39278) 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e0a00640 1 -- 172.21.15.28:0/209049997 learned_addr learned my addr 172.21.15.28:0/209049997 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e1400640 1 --2- 172.21.15.28:0/209049997 >> v2:172.21.15.151:3300/0 conn(0x7f32dc15dba0 0x7f32dc0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059581506 0 0) 0x7f32dc0ce3d0 con 0x7f32dc071c90 2024-03-21T08:42:14.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c4003530 con 0x7f32dc071c90 2024-03-21T08:42:14.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855528753 0 0) 0x7f32dc0ce6b0 con 0x7f32dc0d0140 2024-03-21T08:42:14.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32dc0ce3d0 con 0x7f32dc0d0140 2024-03-21T08:42:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e1400640 1 -- 172.21.15.28:0/209049997 >> v1:172.21.15.151:6789/0 conn(0x7f32dc0d0140 legacy=0x7f32dc0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e1400640 1 -- 172.21.15.28:0/209049997 >> v1:172.21.15.28:6789/0 conn(0x7f32dc071c90 legacy=0x7f32dc0cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.024+0000 7f32e1400640 1 -- 172.21.15.28:0/209049997 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc0ce990 con 0x7f32dc15dba0 2024-03-21T08:42:14.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e1400640 1 --2- 172.21.15.28:0/209049997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc15dba0 0x7f32dc0cddf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f32d00097c0 tx=0x7f32d001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f756e0acf6b2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00b0070 con 0x7f32dc15dba0 2024-03-21T08:42:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d00ab600 con 0x7f32dc15dba0 2024-03-21T08:42:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32dbe00640 1 -- 172.21.15.28:0/209049997 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00ab9d0 con 0x7f32dc15dba0 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- 172.21.15.28:0/209049997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc15dba0 msgr2=0x7f32dc0cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 --2- 172.21.15.28:0/209049997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc15dba0 0x7f32dc0cddf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f32d00097c0 tx=0x7f32d001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- 172.21.15.28:0/209049997 shutdown_connections 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 --2- 172.21.15.28:0/209049997 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc15dba0 0x7f32dc0cddf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- 172.21.15.28:0/209049997 >> 172.21.15.28:0/209049997 conn(0x7f32dc06fa90 msgr2=0x7f32dc06fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- 172.21.15.28:0/209049997 shutdown_connections 2024-03-21T08:42:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- 172.21.15.28:0/209049997 wait complete. 2024-03-21T08:42:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 Processor -- start 2024-03-21T08:42:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.028+0000 7f32e2800640 1 -- start start 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc0d0140 0x7f32dc155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32dc15dba0 0x7f32dc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc0cd870 con 0x7f32dc071c90 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc0cd6f0 con 0x7f32dc0d0140 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc0cd9f0 con 0x7f32dc15dba0 2024-03-21T08:42:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42702/0 (socket says 172.21.15.28:42702) 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 -- 172.21.15.28:0/118197268 learned_addr learned my addr 172.21.15.28:0/118197268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e0a00640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc0d0140 0x7f32dc155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1e00640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32dc15dba0 0x7f32dc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 -- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32dc15dba0 msgr2=0x7f32dc0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32dc15dba0 0x7f32dc0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 -- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc0d0140 msgr2=0x7f32dc155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc0d0140 0x7f32dc155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 -- 172.21.15.28:0/118197268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc156510 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e1400640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f32d00097c0 tx=0x7f32d001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00b0050 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d00bc070 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0006e20 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc156ba0 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32dc157160 con 0x7f32dc071c90 2024-03-21T08:42:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32dc04f410 con 0x7f32dc071c90 2024-03-21T08:42:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.032+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f32d00b5830 con 0x7f32dc071c90 2024-03-21T08:42:14.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.036+0000 7f32d9600640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 0x7f32bc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.036+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f32d0120940 con 0x7f32dc071c90 2024-03-21T08:42:14.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.036+0000 7f32e0a00640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 0x7f32bc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.036+0000 7f32e0a00640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 0x7f32bc069fe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f32dc068620 tx=0x7f32c8007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.040+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f32dc156510 con 0x7f32dc071c90 2024-03-21T08:42:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.212+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32dc1698b0 con 0x7f32dc071c90 2024-03-21T08:42:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.212+0000 7f32d9600640 1 -- 172.21.15.28:0/118197268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f32d0016090 con 0x7f32dc071c90 2024-03-21T08:42:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 msgr2=0x7f32bc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 0x7f32bc069fe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f32dc068620 tx=0x7f32c8007d80 comp rx=0 tx=0).stop 2024-03-21T08:42:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 msgr2=0x7f32dc159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f32d00097c0 tx=0x7f32d001ace0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 shutdown_connections 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32bc067b20 0x7f32bc069fe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32dc15dba0 0x7f32dc0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32dc0d0140 0x7f32dc155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 --2- 172.21.15.28:0/118197268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32dc071c90 0x7f32dc159e70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 >> 172.21.15.28:0/118197268 conn(0x7f32dc06fa90 msgr2=0x7f32dc15e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 shutdown_connections 2024-03-21T08:42:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.216+0000 7f32e2800640 1 -- 172.21.15.28:0/118197268 wait complete. 2024-03-21T08:42:14.236 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi151.front.sepia.ceph.com... 2024-03-21T08:42:14.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:14.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-21T08:42:14.258 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:14.258 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:42:14.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:14.259 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-21T08:42:14.311 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-21T08:42:14.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 Processor -- start 2024-03-21T08:42:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 -- start start 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef815f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 0x7f8ef806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef80cfac0 con 0x7f8ef8099b80 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef80cfda0 con 0x7f8ef80b0b40 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.448+0000 7f8eff400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80d0080 con 0x7f8ef80d0880 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8efca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8ef8099b80 0x7f8ef8099f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39284/0 (socket says 172.21.15.28:39284) 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8efca00640 1 -- 172.21.15.28:0/500902098 learned_addr learned my addr 172.21.15.28:0/500902098 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8efd400640 1 --2- 172.21.15.28:0/500902098 >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 0x7f8ef806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/500902098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7400640 1 -- 172.21.15.28:0/500902098 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170323462 0 0) 0x7f8ef80cfac0 con 0x7f8ef8099b80 2024-03-21T08:42:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7400640 1 -- 172.21.15.28:0/500902098 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003530 con 0x7f8ef8099b80 2024-03-21T08:42:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 -- 172.21.15.28:0/500902098 >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 msgr2=0x7f8ef806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/500902098 >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 0x7f8ef806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 -- 172.21.15.28:0/500902098 >> v1:172.21.15.28:6789/0 conn(0x7f8ef8099b80 legacy=0x7f8ef8099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 -- 172.21.15.28:0/500902098 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef80d0360 con 0x7f8ef80d0880 2024-03-21T08:42:14.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8efd400640 1 --2- 172.21.15.28:0/500902098 >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 0x7f8ef806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:14.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.452+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/500902098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef815f220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097c0 tx=0x7f8ee401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea5cf0f04e2c37e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8ef7400640 1 -- 172.21.15.28:0/500902098 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40b1070 con 0x7f8ef80d0880 2024-03-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8ef7400640 1 -- 172.21.15.28:0/500902098 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee40ac620 con 0x7f8ef80d0880 2024-03-21T08:42:14.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8ef7400640 1 -- 172.21.15.28:0/500902098 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40ac9d0 con 0x7f8ef80d0880 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- 172.21.15.28:0/500902098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 msgr2=0x7f8ef815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- 172.21.15.28:0/500902098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef815f220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40097c0 tx=0x7f8ee401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- 172.21.15.28:0/500902098 shutdown_connections 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- 172.21.15.28:0/500902098 >> v2:172.21.15.151:3300/0 conn(0x7f8ef80b0b40 0x7f8ef806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- 172.21.15.28:0/500902098 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef815f220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- 172.21.15.28:0/500902098 >> 172.21.15.28:0/500902098 conn(0x7f8ef80b3510 msgr2=0x7f8ef80b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- 172.21.15.28:0/500902098 shutdown_connections 2024-03-21T08:42:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- 172.21.15.28:0/500902098 wait complete. 2024-03-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 Processor -- start 2024-03-21T08:42:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- start start 2024-03-21T08:42:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef80b0b40 0x7f8ef8157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef8149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef806d7e0 con 0x7f8ef8099b80 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef80b2d40 con 0x7f8ef80b0b40 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8eff400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef816b050 con 0x7f8ef80d0880 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8efca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8efca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42712/0 (socket says 172.21.15.28:42712) 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8efca00640 1 -- 172.21.15.28:0/793348102 learned_addr learned my addr 172.21.15.28:0/793348102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef80b0b40 0x7f8ef8157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.456+0000 7f8efd400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef8149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 -- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 msgr2=0x7f8ef8149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef8149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 -- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef80b0b40 msgr2=0x7f8ef8157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef80b0b40 0x7f8ef8157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 -- 172.21.15.28:0/793348102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8158730 con 0x7f8ef8099b80 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8efca00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c3c0 tx=0x7f8eec00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec017070 con 0x7f8ef8099b80 2024-03-21T08:42:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec005db0 con 0x7f8ef8099b80 2024-03-21T08:42:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0072f0 con 0x7f8ef8099b80 2024-03-21T08:42:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef8158a10 con 0x7f8ef8099b80 2024-03-21T08:42:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef8159480 con 0x7f8ef8099b80 2024-03-21T08:42:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef804f410 con 0x7f8ef8099b80 2024-03-21T08:42:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8eec01a070 con 0x7f8ef8099b80 2024-03-21T08:42:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 0x7f8ed8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f8eec085740 con 0x7f8ef8099b80 2024-03-21T08:42:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.460+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 0x7f8ed8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.464+0000 7f8ef7e00640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 0x7f8ed8069fe0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40ae000 tx=0x7f8ee40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.464+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8eec0556e0 con 0x7f8ef8099b80 2024-03-21T08:42:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.660+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ef80b43f0 con 0x7f8ef8099b80 2024-03-21T08:42:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.660+0000 7f8ef4c00640 1 -- 172.21.15.28:0/793348102 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f8eec059780 con 0x7f8ef8099b80 2024-03-21T08:42:14.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:14.663 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:42:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 msgr2=0x7f8ed8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 0x7f8ed8069fe0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40ae000 tx=0x7f8ee40a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 msgr2=0x7f8ef80c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c3c0 tx=0x7f8eec00c7c0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 shutdown_connections 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8ed8067b20 0x7f8ed8069fe0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8ef80d0880 0x7f8ef8149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8ef80b0b40 0x7f8ef8157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 --2- 172.21.15.28:0/793348102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ef8099b80 0x7f8ef80c7250 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 >> 172.21.15.28:0/793348102 conn(0x7f8ef80b3510 msgr2=0x7f8ef806fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 shutdown_connections 2024-03-21T08:42:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.664+0000 7f8eff400640 1 -- 172.21.15.28:0/793348102 wait complete. 2024-03-21T08:42:14.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.824+0000 7f987f800640 1 Processor -- start 2024-03-21T08:42:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.824+0000 7f987f800640 1 -- start start 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987f800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98780b5520 0x7f98780cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987f800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98780b34f0 0x7f987815c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987f800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987815bf00 con 0x7f98780af060 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987f800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98780ce6e0 con 0x7f98780b5520 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987f800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98780cdad0 con 0x7f98780b34f0 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98780b5520 0x7f98780cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987ce00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f98780af060 0x7f98780b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42470/0 (socket says 172.21.15.28:42470) 2024-03-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987ce00640 1 -- 172.21.15.28:0/979245989 learned_addr learned my addr 172.21.15.28:0/979245989 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f987d800640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98780b34f0 0x7f987815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 -- 172.21.15.28:0/979245989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98780b34f0 msgr2=0x7f987815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98780b34f0 0x7f987815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877400640 1 -- 172.21.15.28:0/979245989 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863484997 0 0) 0x7f987815bf00 con 0x7f98780af060 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 -- 172.21.15.28:0/979245989 >> v1:172.21.15.151:6789/0 conn(0x7f98780af060 legacy=0x7f98780b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 -- 172.21.15.28:0/979245989 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98780cddb0 con 0x7f98780b5520 2024-03-21T08:42:14.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877e00640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f98780cd380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f98640097c0 tx=0x7f986401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f421043b799f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877400640 1 -- 172.21.15.28:0/979245989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640b0070 con 0x7f98780b5520 2024-03-21T08:42:14.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877400640 1 -- 172.21.15.28:0/979245989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98640ab600 con 0x7f98780b5520 2024-03-21T08:42:14.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.828+0000 7f9877400640 1 -- 172.21.15.28:0/979245989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640ab9d0 con 0x7f98780b5520 2024-03-21T08:42:14.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- 172.21.15.28:0/979245989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 msgr2=0x7f98780cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f98780cd380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f98640097c0 tx=0x7f986401c380 comp rx=0 tx=0).stop 2024-03-21T08:42:14.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- 172.21.15.28:0/979245989 shutdown_connections 2024-03-21T08:42:14.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98780b34f0 0x7f987815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- 172.21.15.28:0/979245989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f98780cd380 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- 172.21.15.28:0/979245989 >> 172.21.15.28:0/979245989 conn(0x7f98780cc230 msgr2=0x7f98780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:14.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- 172.21.15.28:0/979245989 shutdown_connections 2024-03-21T08:42:14.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- 172.21.15.28:0/979245989 wait complete. 2024-03-21T08:42:14.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 Processor -- start 2024-03-21T08:42:14.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- start start 2024-03-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 0x7f98780ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f987814b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f987806ea40 0x7f987814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987815d0d0 con 0x7f98780b5520 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987815cf50 con 0x7f98780b34f0 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987f800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987815d250 con 0x7f987806ea40 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f9877e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f987814b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987ce00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 0x7f98780ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f9877e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f987814b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42738/0 (socket says 172.21.15.28:42738) 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f9877e00640 1 -- 172.21.15.28:0/3205993450 learned_addr learned my addr 172.21.15.28:0/3205993450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.832+0000 7f987d800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f987806ea40 0x7f987814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 -- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f987806ea40 msgr2=0x7f987814a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f987806ea40 0x7f987814a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 -- 172.21.15.28:0/3205993450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 msgr2=0x7f987814b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f987814b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 -- 172.21.15.28:0/3205993450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98780adfe0 con 0x7f98780b34f0 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987ce00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 0x7f98780ae7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f986c00c430 tx=0x7f986c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c0074e0 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c0047f0 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c004be0 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987815e930 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98780ae2a0 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987804f410 con 0x7f98780b34f0 2024-03-21T08:42:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f986c011070 con 0x7f98780b34f0 2024-03-21T08:42:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 0x7f9858069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f986c085ee0 con 0x7f98780b34f0 2024-03-21T08:42:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.836+0000 7f9877e00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 0x7f9858069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.840+0000 7f9877e00640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 0x7f9858069fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9864019fb0 tx=0x7f98640a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:14.844+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f986c055eb0 con 0x7f98780b34f0 2024-03-21T08:42:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.004+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98780008e0 con 0x7f98780b34f0 2024-03-21T08:42:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.004+0000 7f9874c00640 1 -- 172.21.15.28:0/3205993450 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f986c059f50 con 0x7f98780b34f0 2024-03-21T08:42:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:15.007 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 msgr2=0x7f9858069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 0x7f9858069fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9864019fb0 tx=0x7f98640a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 msgr2=0x7f98780ae7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 0x7f98780ae7c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f986c00c430 tx=0x7f986c00c8f0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 shutdown_connections 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9858067b20 0x7f9858069fe0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f987806ea40 0x7f987814a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98780b5520 0x7f987814b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 --2- 172.21.15.28:0/3205993450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98780b34f0 0x7f98780ae7c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 >> 172.21.15.28:0/3205993450 conn(0x7f98780cc230 msgr2=0x7f987815a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 shutdown_connections 2024-03-21T08:42:15.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.008+0000 7f987f800640 1 -- 172.21.15.28:0/3205993450 wait complete. 2024-03-21T08:42:15.028 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:41:05.747726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:41:06.377508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-21T08:41:10.058077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-21T08:42:15.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 Processor -- start 2024-03-21T08:42:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 -- start start 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2e380ccb70 0x7f2e380cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e3806b170 0x7f2e3806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380d0760 con 0x7f2e3815e020 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e380ce7b0 con 0x7f2e3806b170 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e3d200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e380cfc90 con 0x7f2e380ccb70 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e3806b170 0x7f2e3806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e3806b170 0x7f2e3806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42754/0 (socket says 172.21.15.28:42754) 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 -- 172.21.15.28:0/2735160183 learned_addr learned my addr 172.21.15.28:0/2735160183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e35400640 1 --2- 172.21.15.28:0/2735160183 >> v2:172.21.15.151:3300/0 conn(0x7f2e380ccb70 0x7f2e380cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 -- 172.21.15.28:0/2735160183 >> v2:172.21.15.151:3300/0 conn(0x7f2e380ccb70 msgr2=0x7f2e380cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 --2- 172.21.15.28:0/2735160183 >> v2:172.21.15.151:3300/0 conn(0x7f2e380ccb70 0x7f2e380cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 -- 172.21.15.28:0/2735160183 >> v1:172.21.15.151:6789/0 conn(0x7f2e3815e020 legacy=0x7f2e380cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.172+0000 7f2e36800640 1 -- 172.21.15.28:0/2735160183 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380cfec0 con 0x7f2e3806b170 2024-03-21T08:42:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e36800640 1 --2- 172.21.15.28:0/2735160183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e3806b170 0x7f2e3806b5c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009e50 tx=0x7f2e3001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45775f042a6aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e34a00640 1 -- 172.21.15.28:0/2735160183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300a56c0 con 0x7f2e3806b170 2024-03-21T08:42:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e34a00640 1 -- 172.21.15.28:0/2735160183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e300a5860 con 0x7f2e3806b170 2024-03-21T08:42:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e34a00640 1 -- 172.21.15.28:0/2735160183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300a5c30 con 0x7f2e3806b170 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- 172.21.15.28:0/2735160183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e3806b170 msgr2=0x7f2e3806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 --2- 172.21.15.28:0/2735160183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e3806b170 0x7f2e3806b5c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009e50 tx=0x7f2e3001c9f0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- 172.21.15.28:0/2735160183 shutdown_connections 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 --2- 172.21.15.28:0/2735160183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e3806b170 0x7f2e3806b5c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 --2- 172.21.15.28:0/2735160183 >> v2:172.21.15.151:3300/0 conn(0x7f2e380ccb70 0x7f2e380cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- 172.21.15.28:0/2735160183 >> 172.21.15.28:0/2735160183 conn(0x7f2e38099a30 msgr2=0x7f2e38099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- 172.21.15.28:0/2735160183 shutdown_connections 2024-03-21T08:42:15.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- 172.21.15.28:0/2735160183 wait complete. 2024-03-21T08:42:15.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 Processor -- start 2024-03-21T08:42:15.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.176+0000 7f2e3d200640 1 -- start start 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2e3806b170 0x7f2e38145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e3815e020 0x7f2e38159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815f410 con 0x7f2e380ccb70 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815e790 con 0x7f2e3815e020 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815f590 con 0x7f2e3806b170 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42764/0 (socket says 172.21.15.28:42764) 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 -- 172.21.15.28:0/3859262430 learned_addr learned my addr 172.21.15.28:0/3859262430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e36800640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e3815e020 0x7f2e38159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35e00640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2e3806b170 0x7f2e38145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 -- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2e3806b170 msgr2=0x7f2e38145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2e3806b170 0x7f2e38145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 -- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e3815e020 msgr2=0x7f2e38159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e3815e020 0x7f2e38159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 -- 172.21.15.28:0/3859262430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380673c0 con 0x7f2e380ccb70 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e35400640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e960 tx=0x7f2e2400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24017070 con 0x7f2e380ccb70 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2400ca00 con 0x7f2e380ccb70 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400cdb0 con 0x7f2e380ccb70 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e380c7350 con 0x7f2e380ccb70 2024-03-21T08:42:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e380c7070 con 0x7f2e380ccb70 2024-03-21T08:42:15.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.180+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3809eb60 con 0x7f2e380ccb70 2024-03-21T08:42:15.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.184+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2e2401a070 con 0x7f2e380ccb70 2024-03-21T08:42:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.184+0000 7f2e2e000640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 0x7f2e1806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.184+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2e24086750 con 0x7f2e380ccb70 2024-03-21T08:42:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.184+0000 7f2e35e00640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 0x7f2e1806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.184+0000 7f2e35e00640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 0x7f2e1806a010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e28001810 tx=0x7f2e28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.188+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2e240565d0 con 0x7f2e380ccb70 2024-03-21T08:42:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e3804f4f0 con 0x7f2e380ccb70 2024-03-21T08:42:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e2e000640 1 -- 172.21.15.28:0/3859262430 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e2405a670 con 0x7f2e380ccb70 2024-03-21T08:42:15.353 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 msgr2=0x7f2e1806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 0x7f2e1806a010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e28001810 tx=0x7f2e28009040 comp rx=0 tx=0).stop 2024-03-21T08:42:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 msgr2=0x7f2e38159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400e960 tx=0x7f2e2400ee20 comp rx=0 tx=0).stop 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 shutdown_connections 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2e3815e020 0x7f2e38159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e380ccb70 0x7f2e38159020 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2e18067b50 0x7f2e1806a010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 --2- 172.21.15.28:0/3859262430 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2e3806b170 0x7f2e38145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 >> 172.21.15.28:0/3859262430 conn(0x7f2e38099a30 msgr2=0x7f2e3815bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 shutdown_connections 2024-03-21T08:42:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.352+0000 7f2e3d200640 1 -- 172.21.15.28:0/3859262430 wait complete. 2024-03-21T08:42:15.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 Processor -- start 2024-03-21T08:42:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 -- start start 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f71240aedd0 0x7f712415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71240b2de0 0x7f71240ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f71240727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71240cbdf0 con 0x7f71240b2de0 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71240cee20 con 0x7f71240aedd0 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f712a000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71240cf100 con 0x7f7124072310 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f711be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71240b2de0 0x7f71240ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f7123400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f71240aedd0 0x7f712415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f711be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71240b2de0 0x7f71240ce5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42774/0 (socket says 172.21.15.28:42774) 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f7123e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f71240727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.516+0000 7f711be00640 1 -- 172.21.15.28:0/1849061569 learned_addr learned my addr 172.21.15.28:0/1849061569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 -- 172.21.15.28:0/1849061569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 msgr2=0x7f71240727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 --2- 172.21.15.28:0/1849061569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f71240727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 -- 172.21.15.28:0/1849061569 >> v2:172.21.15.151:3300/0 conn(0x7f71240aedd0 msgr2=0x7f712415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 --2- 172.21.15.28:0/1849061569 >> v2:172.21.15.151:3300/0 conn(0x7f71240aedd0 0x7f712415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 -- 172.21.15.28:0/1849061569 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240cf3e0 con 0x7f71240b2de0 2024-03-21T08:42:15.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f711be00640 1 --2- 172.21.15.28:0/1849061569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240b2de0 0x7f71240ce5f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f712406efd0 tx=0x7f710c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608a729540a90957 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f7122a00640 1 -- 172.21.15.28:0/1849061569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0b0070 con 0x7f71240b2de0 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f7122a00640 1 -- 172.21.15.28:0/1849061569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c0ab600 con 0x7f71240b2de0 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f7122a00640 1 -- 172.21.15.28:0/1849061569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0ab9d0 con 0x7f71240b2de0 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f712a000640 1 -- 172.21.15.28:0/1849061569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240b2de0 msgr2=0x7f71240ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.520+0000 7f712a000640 1 --2- 172.21.15.28:0/1849061569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240b2de0 0x7f71240ce5f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f712406efd0 tx=0x7f710c01c520 comp rx=0 tx=0).stop 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/1849061569 shutdown_connections 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- 172.21.15.28:0/1849061569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f71240727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- 172.21.15.28:0/1849061569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240b2de0 0x7f71240ce5f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- 172.21.15.28:0/1849061569 >> v2:172.21.15.151:3300/0 conn(0x7f71240aedd0 0x7f712415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/1849061569 >> 172.21.15.28:0/1849061569 conn(0x7f71240674f0 msgr2=0x7f712406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/1849061569 shutdown_connections 2024-03-21T08:42:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/1849061569 wait complete. 2024-03-21T08:42:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 Processor -- start 2024-03-21T08:42:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- start start 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f7124152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71240b2de0 0x7f71241414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71240cc3e0 con 0x7f71240aedd0 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71240cc260 con 0x7f71240b2de0 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71240cc560 con 0x7f7124072310 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42784/0 (socket says 172.21.15.28:42784) 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 -- 172.21.15.28:0/428085433 learned_addr learned my addr 172.21.15.28:0/428085433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f7123400640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f7124152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f7123e00640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71240b2de0 0x7f71241414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 -- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 msgr2=0x7f7124152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f7124152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 -- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71240b2de0 msgr2=0x7f71241414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71240b2de0 0x7f71241414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 -- 172.21.15.28:0/428085433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71241f7610 con 0x7f71240aedd0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f7123400640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f7124152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711be00640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f710c01abd0 tx=0x7f710c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0b0040 con 0x7f71240aedd0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c0b4620 con 0x7f71240aedd0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c0b4a10 con 0x7f71240aedd0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71241f78f0 con 0x7f71240aedd0 2024-03-21T08:42:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.524+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71241f7eb0 con 0x7f71240aedd0 2024-03-21T08:42:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7124073430 con 0x7f71240aedd0 2024-03-21T08:42:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f710c0ab4c0 con 0x7f71240aedd0 2024-03-21T08:42:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f711b400640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 0x7f710006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f710c11ff90 con 0x7f71240aedd0 2024-03-21T08:42:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f7123400640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 0x7f710006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.528+0000 7f7123400640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 0x7f710006a010 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7114009bb0 tx=0x7f7114008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.532+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f710c016110 con 0x7f71240aedd0 2024-03-21T08:42:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.696+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f712404f4f0 con 0x7f71240aedd0 2024-03-21T08:42:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.696+0000 7f711b400640 1 -- 172.21.15.28:0/428085433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f710c0f0580 con 0x7f71240aedd0 2024-03-21T08:42:15.698 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 msgr2=0x7f710006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 0x7f710006a010 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7114009bb0 tx=0x7f7114008040 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 msgr2=0x7f7124140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f710c01abd0 tx=0x7f710c0192c0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 shutdown_connections 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f71240b2de0 0x7f71241414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71240aedd0 0x7f7124140f90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7100067b50 0x7f710006a010 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 --2- 172.21.15.28:0/428085433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7124072310 0x7f7124152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 >> 172.21.15.28:0/428085433 conn(0x7f71240674f0 msgr2=0x7f7124142580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 shutdown_connections 2024-03-21T08:42:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.700+0000 7f712a000640 1 -- 172.21.15.28:0/428085433 wait complete. 2024-03-21T08:42:15.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:15.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 Processor -- start 2024-03-21T08:42:15.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 -- start start 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f481c0b4420 0x7f481c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c15bf90 con 0x7f481c0cd300 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c15b1d0 con 0x7f481c0b4800 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4824400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c15b4b0 con 0x7f481c0b4420 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4821a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f481c0b4420 0x7f481c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4822400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f481c0cd300 0x7f481c0b2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39298/0 (socket says 172.21.15.28:39298) 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4821a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f481c0b4420 0x7f481c071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42786/0 (socket says 172.21.15.28:42786) 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4822400640 1 -- 172.21.15.28:0/916349350 learned_addr learned my addr 172.21.15.28:0/916349350 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4821a00640 1 -- 172.21.15.28:0/916349350 >> v1:172.21.15.151:6789/0 conn(0x7f481c0b4800 legacy=0x7f481c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4821a00640 1 -- 172.21.15.28:0/916349350 >> v1:172.21.15.28:6789/0 conn(0x7f481c0cd300 legacy=0x7f481c0b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f4821a00640 1 -- 172.21.15.28:0/916349350 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c15b790 con 0x7f481c0b4420 2024-03-21T08:42:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.864+0000 7f481be00640 1 -- 172.21.15.28:0/916349350 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044742383 0 0) 0x7f481c15bf90 con 0x7f481c0cd300 2024-03-21T08:42:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4821a00640 1 --2- 172.21.15.28:0/916349350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0b4420 0x7f481c071a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f481000ba60 tx=0x7f481001eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d344b00bb7f470ab server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f481be00640 1 -- 172.21.15.28:0/916349350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100b4070 con 0x7f481c0b4420 2024-03-21T08:42:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f481be00640 1 -- 172.21.15.28:0/916349350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48100a7b20 con 0x7f481c0b4420 2024-03-21T08:42:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f481be00640 1 -- 172.21.15.28:0/916349350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100af440 con 0x7f481c0b4420 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 -- 172.21.15.28:0/916349350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0b4420 msgr2=0x7f481c071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 --2- 172.21.15.28:0/916349350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0b4420 0x7f481c071a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f481000ba60 tx=0x7f481001eb40 comp rx=0 tx=0).stop 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 -- 172.21.15.28:0/916349350 shutdown_connections 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 --2- 172.21.15.28:0/916349350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0b4420 0x7f481c071a80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 -- 172.21.15.28:0/916349350 >> 172.21.15.28:0/916349350 conn(0x7f481c0674f0 msgr2=0x7f481c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 -- 172.21.15.28:0/916349350 shutdown_connections 2024-03-21T08:42:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.868+0000 7f4824400640 1 -- 172.21.15.28:0/916349350 wait complete. 2024-03-21T08:42:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 Processor -- start 2024-03-21T08:42:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- start start 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f481c0b4420 0x7f481c142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f481c0cd300 0x7f481c0c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c06de30 con 0x7f481c0c53a0 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c06dcb0 con 0x7f481c0b4420 2024-03-21T08:42:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c06dfb0 con 0x7f481c0cd300 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4821a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f481c0b4420 0x7f481c142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42798/0 (socket says 172.21.15.28:42798) 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 -- 172.21.15.28:0/1252290720 learned_addr learned my addr 172.21.15.28:0/1252290720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4821000640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f481c0cd300 0x7f481c0c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 -- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f481c0cd300 msgr2=0x7f481c0c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f481c0cd300 0x7f481c0c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 -- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f481c0b4420 msgr2=0x7f481c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f481c0b4420 0x7f481c142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 -- 172.21.15.28:0/1252290720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c1505a0 con 0x7f481c0c53a0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4822400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f481400c780 tx=0x7f481400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814018070 con 0x7f481c0c53a0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814004d70 con 0x7f481c0c53a0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481400a880 con 0x7f481c0c53a0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c150880 con 0x7f481c0c53a0 2024-03-21T08:42:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c150e40 con 0x7f481c0c53a0 2024-03-21T08:42:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.872+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c073430 con 0x7f481c0c53a0 2024-03-21T08:42:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.876+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f481401b070 con 0x7f481c0c53a0 2024-03-21T08:42:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.876+0000 7f4819600640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 0x7f47fc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.876+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f4814085850 con 0x7f481c0c53a0 2024-03-21T08:42:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.876+0000 7f4821a00640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 0x7f47fc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:15.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.876+0000 7f4821a00640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 0x7f47fc069fe0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f48100b1000 tx=0x7f4810008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:15.880+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f48140557f0 con 0x7f481c0c53a0 2024-03-21T08:42:16.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.040+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f481c06b660 con 0x7f481c0c53a0 2024-03-21T08:42:16.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.040+0000 7f4819600640 1 -- 172.21.15.28:0/1252290720 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4814059890 con 0x7f481c0c53a0 2024-03-21T08:42:16.044 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 msgr2=0x7f47fc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 0x7f47fc069fe0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f48100b1000 tx=0x7f4810008040 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 msgr2=0x7f481c153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f481400c780 tx=0x7f481400cc40 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 shutdown_connections 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f481c0c53a0 0x7f481c153160 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f47fc067b20 0x7f47fc069fe0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f481c0cd300 0x7f481c0c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 --2- 172.21.15.28:0/1252290720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f481c0b4420 0x7f481c142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 >> 172.21.15.28:0/1252290720 conn(0x7f481c0674f0 msgr2=0x7f481c148f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 shutdown_connections 2024-03-21T08:42:16.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.044+0000 7f4824400640 1 -- 172.21.15.28:0/1252290720 wait complete. 2024-03-21T08:42:16.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-21T08:42:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.208+0000 7fceb0000640 1 Processor -- start 2024-03-21T08:42:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.208+0000 7fceb0000640 1 -- start start 2024-03-21T08:42:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceb0000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcea80b4420 0x7fcea8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceb0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 0x7fcea8071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceb0000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea815bf90 con 0x7fcea80ccbf0 2024-03-21T08:42:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceb0000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea815b1d0 con 0x7fcea80b4420 2024-03-21T08:42:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceb0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea815b4b0 con 0x7fcea80b4800 2024-03-21T08:42:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcea80b4420 0x7fcea8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceacc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 0x7fcea8071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcea80b4420 0x7fcea8071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35710/0 (socket says 172.21.15.28:35710) 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 -- 172.21.15.28:0/2479734540 learned_addr learned my addr 172.21.15.28:0/2479734540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceae000640 1 --1- 172.21.15.28:0/2479734540 >> v1:172.21.15.151:6789/0 conn(0x7fcea80ccbf0 0x7fcea80cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42502/0 (socket says 172.21.15.28:42502) 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fceacc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 0x7fcea8071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:51834/0 (socket says 172.21.15.28:51834) 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 -- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 msgr2=0x7fcea8071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 --2- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 0x7fcea8071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 -- 172.21.15.28:0/2479734540 >> v1:172.21.15.151:6789/0 conn(0x7fcea80ccbf0 legacy=0x7fcea80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.212+0000 7fcead600640 1 -- 172.21.15.28:0/2479734540 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea815b790 con 0x7fcea80b4420 2024-03-21T08:42:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fcead600640 1 --2- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea80b4420 0x7fcea8071a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009870 tx=0x7fce9c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69017489190f7e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fcea7e00640 1 -- 172.21.15.28:0/2479734540 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0ab430 con 0x7fcea80b4420 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fcea7e00640 1 -- 172.21.15.28:0/2479734540 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c0b0070 con 0x7fcea80b4420 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fcea7e00640 1 -- 172.21.15.28:0/2479734540 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0ab7f0 con 0x7fcea80b4420 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea80b4420 msgr2=0x7fcea8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 --2- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea80b4420 0x7fcea8071a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009870 tx=0x7fce9c01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- 172.21.15.28:0/2479734540 shutdown_connections 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 --2- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4800 0x7fcea8071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 --2- 172.21.15.28:0/2479734540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea80b4420 0x7fcea8071a80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- 172.21.15.28:0/2479734540 >> 172.21.15.28:0/2479734540 conn(0x7fcea80674f0 msgr2=0x7fcea806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- 172.21.15.28:0/2479734540 shutdown_connections 2024-03-21T08:42:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- 172.21.15.28:0/2479734540 wait complete. 2024-03-21T08:42:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 Processor -- start 2024-03-21T08:42:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.216+0000 7fceb0000640 1 -- start start 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea80ccbf0 0x7fcea8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea815a040 0x7fcea8149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80b3670 con 0x7fcea80ccbf0 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80b34f0 con 0x7fcea815a040 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80b37f0 con 0x7fcea80b4420 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea80ccbf0 0x7fcea8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:51838/0 (socket says 172.21.15.28:51838) 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 -- 172.21.15.28:0/1119312671 learned_addr learned my addr 172.21.15.28:0/1119312671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceae000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea815a040 0x7fcea8149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 -- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea815a040 msgr2=0x7fcea8149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea815a040 0x7fcea8149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 -- 172.21.15.28:0/1119312671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea80ccbf0 msgr2=0x7fcea8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea80ccbf0 0x7fcea8159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 -- 172.21.15.28:0/1119312671 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea814d650 con 0x7fcea80b4420 2024-03-21T08:42:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcead600640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0ad040 tx=0x7fce9c019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0ba070 con 0x7fcea80b4420 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c0abe40 con 0x7fcea80b4420 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0b4660 con 0x7fcea80b4420 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea81fa4a0 con 0x7fcea80b4420 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea81faa60 con 0x7fcea80b4420 2024-03-21T08:42:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.220+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea815a450 con 0x7fcea80b4420 2024-03-21T08:42:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.224+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fce9c0bd070 con 0x7fcea80b4420 2024-03-21T08:42:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.224+0000 7fcea5600640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 0x7fce8806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.224+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fce9c1202a0 con 0x7fcea80b4420 2024-03-21T08:42:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.224+0000 7fceacc00640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 0x7fce8806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.224+0000 7fceacc00640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 0x7fce8806a010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcea8167820 tx=0x7fce94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.228+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcea815a450 con 0x7fcea80b4420 2024-03-21T08:42:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.392+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcea8073430 con 0x7fcea80b4420 2024-03-21T08:42:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.392+0000 7fcea5600640 1 -- 172.21.15.28:0/1119312671 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fce9c016150 con 0x7fcea80b4420 2024-03-21T08:42:16.394 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.392+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 msgr2=0x7fce8806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.392+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 0x7fce8806a010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcea8167820 tx=0x7fce94009190 comp rx=0 tx=0).stop 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.392+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 msgr2=0x7fcea814d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0ad040 tx=0x7fce9c019240 comp rx=0 tx=0).stop 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 shutdown_connections 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcea815a040 0x7fcea8149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea80ccbf0 0x7fcea8159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce88067b50 0x7fce8806a010 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 --2- 172.21.15.28:0/1119312671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcea80b4420 0x7fcea814d110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 >> 172.21.15.28:0/1119312671 conn(0x7fcea80674f0 msgr2=0x7fcea814d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 shutdown_connections 2024-03-21T08:42:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.396+0000 7fceb0000640 1 -- 172.21.15.28:0/1119312671 wait complete. 2024-03-21T08:42:16.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:16.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 Processor -- start 2024-03-21T08:42:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 -- start start 2024-03-21T08:42:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721015e0c0 0x7f721006cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7210099b80 0x7f7210099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100cc1d0 con 0x7f72100ccc70 2024-03-21T08:42:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100d1730 con 0x7f7210099b80 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7215600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721009a520 con 0x7f721015e0c0 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7210099b80 0x7f7210099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7210099b80 0x7f7210099fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42814/0 (socket says 172.21.15.28:42814) 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 -- 172.21.15.28:0/958428294 learned_addr learned my addr 172.21.15.28:0/958428294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f720ea00640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721015e0c0 0x7f721006cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 -- 172.21.15.28:0/958428294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721015e0c0 msgr2=0x7f721006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721015e0c0 0x7f721006cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 -- 172.21.15.28:0/958428294 >> v1:172.21.15.151:6789/0 conn(0x7f72100ccc70 legacy=0x7f72100cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.556+0000 7f7207e00640 1 -- 172.21.15.28:0/958428294 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721006d450 con 0x7f7210099b80 2024-03-21T08:42:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7207e00640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7210099b80 0x7f7210099fe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2720ed492b8948 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f720e000640 1 -- 172.21.15.28:0/958428294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80a95f0 con 0x7f7210099b80 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f720e000640 1 -- 172.21.15.28:0/958428294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80a9790 con 0x7f7210099b80 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f720e000640 1 -- 172.21.15.28:0/958428294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80a9b60 con 0x7f7210099b80 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 -- 172.21.15.28:0/958428294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7210099b80 msgr2=0x7f7210099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7210099b80 0x7f7210099fe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f801c530 comp rx=0 tx=0).stop 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 -- 172.21.15.28:0/958428294 shutdown_connections 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7210099b80 0x7f7210099fe0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 --2- 172.21.15.28:0/958428294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721015e0c0 0x7f721006cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 -- 172.21.15.28:0/958428294 >> 172.21.15.28:0/958428294 conn(0x7f72100b34d0 msgr2=0x7f72100b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 -- 172.21.15.28:0/958428294 shutdown_connections 2024-03-21T08:42:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.560+0000 7f7215600640 1 -- 172.21.15.28:0/958428294 wait complete. 2024-03-21T08:42:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 Processor -- start 2024-03-21T08:42:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- start start 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72100ccc70 0x7f72100b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 0x7f721015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100d10d0 con 0x7f721015e0c0 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721015e4a0 con 0x7f72100ccc70 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721016ac90 con 0x7f7210099b80 2024-03-21T08:42:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 0x7f721015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:51852/0 (socket says 172.21.15.28:51852) 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 -- 172.21.15.28:0/1191050705 learned_addr learned my addr 172.21.15.28:0/1191050705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7207e00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72100ccc70 0x7f72100b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 -- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72100ccc70 msgr2=0x7f72100b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72100ccc70 0x7f72100b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 -- 172.21.15.28:0/1191050705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 msgr2=0x7f721015add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 0x7f721015add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 -- 172.21.15.28:0/1191050705 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210154070 con 0x7f7210099b80 2024-03-21T08:42:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720f400640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 0x7f721015add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:16.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f720ea00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f720000e8a0 tx=0x7f720000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200017070 con 0x7f7210099b80 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720000c940 con 0x7f7210099b80 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720000ccf0 con 0x7f7210099b80 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7210154350 con 0x7f7210099b80 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.564+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100c5580 con 0x7f7210099b80 2024-03-21T08:42:16.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721004f410 con 0x7f7210099b80 2024-03-21T08:42:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f720001a070 con 0x7f7210099b80 2024-03-21T08:42:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7206a00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 0x7f71ec069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f7200086500 con 0x7f7210099b80 2024-03-21T08:42:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7207e00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 0x7f71ec069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.568+0000 7f7207e00640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 0x7f71ec069fe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f71f80ae000 tx=0x7f71f80b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.572+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f72000564a0 con 0x7f7210099b80 2024-03-21T08:42:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.768+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72100b7820 con 0x7f7210099b80 2024-03-21T08:42:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.768+0000 7f7206a00640 1 -- 172.21.15.28:0/1191050705 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f720005a540 con 0x7f7210099b80 2024-03-21T08:42:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:16.772 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:42:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:42:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 msgr2=0x7f71ec069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 0x7f71ec069fe0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f71f80ae000 tx=0x7f71f80b3040 comp rx=0 tx=0).stop 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 msgr2=0x7f7210153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f720000e8a0 tx=0x7f720000eca0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 shutdown_connections 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721015e0c0 0x7f721015add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f72100ccc70 0x7f72100b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71ec067b20 0x7f71ec069fe0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 --2- 172.21.15.28:0/1191050705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7210099b80 0x7f7210153810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 >> 172.21.15.28:0/1191050705 conn(0x7f72100b34d0 msgr2=0x7f721015cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 shutdown_connections 2024-03-21T08:42:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.772+0000 7f7215600640 1 -- 172.21.15.28:0/1191050705 wait complete. 2024-03-21T08:42:16.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:16.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 Processor -- start 2024-03-21T08:42:16.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 -- start start 2024-03-21T08:42:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fec700b5520 0x7fec700cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7015bf00 con 0x7fec700b34f0 2024-03-21T08:42:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec700cdad0 con 0x7fec700af060 2024-03-21T08:42:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.932+0000 7fec75e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec700ce6e0 con 0x7fec700b5520 2024-03-21T08:42:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fec700b34f0 0x7fec7015c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39308/0 (socket says 172.21.15.28:39308) 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6f400640 1 -- 172.21.15.28:0/2764807957 learned_addr learned my addr 172.21.15.28:0/2764807957 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec67e00640 1 --2- 172.21.15.28:0/2764807957 >> v2:172.21.15.151:3300/0 conn(0x7fec700b5520 0x7fec700cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663767438 0 0) 0x7fec7015bf00 con 0x7fec700b34f0 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54003530 con 0x7fec700b34f0 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec67e00640 1 -- 172.21.15.28:0/2764807957 >> v1:172.21.15.151:6789/0 conn(0x7fec700af060 legacy=0x7fec700b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188014393 0 0) 0x7fec700cdad0 con 0x7fec700af060 2024-03-21T08:42:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec67e00640 1 -- 172.21.15.28:0/2764807957 >> v1:172.21.15.28:6789/0 conn(0x7fec700b34f0 legacy=0x7fec7015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec67e00640 1 -- 172.21.15.28:0/2764807957 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700cddb0 con 0x7fec700b5520 2024-03-21T08:42:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec67e00640 1 --2- 172.21.15.28:0/2764807957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec700b5520 0x7fec700cd380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec700ce860 tx=0x7fec5801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1315421a64f90063 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580b0070 con 0x7fec700b5520 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec580ab600 con 0x7fec700b5520 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.936+0000 7fec6e000640 1 -- 172.21.15.28:0/2764807957 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec580ab9d0 con 0x7fec700b5520 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- 172.21.15.28:0/2764807957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec700b5520 msgr2=0x7fec700cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 --2- 172.21.15.28:0/2764807957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec700b5520 0x7fec700cd380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec700ce860 tx=0x7fec5801c410 comp rx=0 tx=0).stop 2024-03-21T08:42:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- 172.21.15.28:0/2764807957 shutdown_connections 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 --2- 172.21.15.28:0/2764807957 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec700b5520 0x7fec700cd380 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- 172.21.15.28:0/2764807957 >> 172.21.15.28:0/2764807957 conn(0x7fec700cc230 msgr2=0x7fec700cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- 172.21.15.28:0/2764807957 shutdown_connections 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- 172.21.15.28:0/2764807957 wait complete. 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 Processor -- start 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- start start 2024-03-21T08:42:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec700b5520 0x7fec700ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 0x7fec7015e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015d2a0 con 0x7fec700b34f0 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015d120 con 0x7fec7014be40 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec75e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015d420 con 0x7fec700b5520 2024-03-21T08:42:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42838/0 (socket says 172.21.15.28:42838) 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 -- 172.21.15.28:0/4009651158 learned_addr learned my addr 172.21.15.28:0/4009651158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec67e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec700b5520 0x7fec700ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6f400640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 0x7fec7015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 -- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec700b5520 msgr2=0x7fec700ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec700b5520 0x7fec700ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 -- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 msgr2=0x7fec7015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 0x7fec7015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6ea00640 1 -- 172.21.15.28:0/4009651158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7015bf00 con 0x7fec700b34f0 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.940+0000 7fec6f400640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 0x7fec7015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec6ea00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c4d0 tx=0x7fec6000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60017070 con 0x7fec700b34f0 2024-03-21T08:42:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec60005480 con 0x7fec700b34f0 2024-03-21T08:42:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60003840 con 0x7fec700b34f0 2024-03-21T08:42:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7006e8b0 con 0x7fec700b34f0 2024-03-21T08:42:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec701fa3e0 con 0x7fec700b34f0 2024-03-21T08:42:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7004f410 con 0x7fec700b34f0 2024-03-21T08:42:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fec6001a070 con 0x7fec700b34f0 2024-03-21T08:42:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 0x7fec4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fec60085720 con 0x7fec700b34f0 2024-03-21T08:42:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.944+0000 7fec67e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 0x7fec4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.948+0000 7fec67e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 0x7fec4c069fe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec580ad000 tx=0x7fec58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:16.948+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fec600556f0 con 0x7fec700b34f0 2024-03-21T08:42:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.112+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec700ccd50 con 0x7fec700b34f0 2024-03-21T08:42:17.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.112+0000 7fec67400640 1 -- 172.21.15.28:0/4009651158 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7fec60059790 con 0x7fec700b34f0 2024-03-21T08:42:17.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:17.116 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:17.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 msgr2=0x7fec4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 0x7fec4c069fe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec580ad000 tx=0x7fec58008040 comp rx=0 tx=0).stop 2024-03-21T08:42:17.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 msgr2=0x7fec7006e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fec6000c4d0 tx=0x7fec6000c990 comp rx=0 tx=0).stop 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 shutdown_connections 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec7014be40 0x7fec7015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec4c067b20 0x7fec4c069fe0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec700b5520 0x7fec700ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 --2- 172.21.15.28:0/4009651158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec700b34f0 0x7fec7006e2d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 >> 172.21.15.28:0/4009651158 conn(0x7fec700cc230 msgr2=0x7fec70156690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 shutdown_connections 2024-03-21T08:42:17.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.116+0000 7fec75e00640 1 -- 172.21.15.28:0/4009651158 wait complete. 2024-03-21T08:42:17.137 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-21T08:42:17.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.276+0000 7f3577e00640 1 Processor -- start 2024-03-21T08:42:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.276+0000 7f3577e00640 1 -- start start 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3577e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 0x7f35700b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3577e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357015bf00 con 0x7f35700b5520 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3577e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700cdad0 con 0x7f35700af060 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3577e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b3a30 con 0x7f35700cd380 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3574a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35700b5520 0x7f357015c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39320/0 (socket says 172.21.15.28:39320) 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3574a00640 1 -- 172.21.15.28:0/3521433278 learned_addr learned my addr 172.21.15.28:0/3521433278 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3575e00640 1 --2- 172.21.15.28:0/3521433278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 0x7f35700b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942748810 0 0) 0x7f357015bf00 con 0x7f35700b5520 2024-03-21T08:42:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003530 con 0x7f35700b5520 2024-03-21T08:42:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617285418 0 0) 0x7f35700cdad0 con 0x7f35700af060 2024-03-21T08:42:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f357015bf00 con 0x7f35700af060 2024-03-21T08:42:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3575e00640 1 -- 172.21.15.28:0/3521433278 >> v1:172.21.15.151:6789/0 conn(0x7f35700af060 legacy=0x7f35700b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3575e00640 1 -- 172.21.15.28:0/3521433278 >> v1:172.21.15.28:6789/0 conn(0x7f35700b5520 legacy=0x7f357015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3575e00640 1 -- 172.21.15.28:0/3521433278 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35700cddb0 con 0x7f35700cd380 2024-03-21T08:42:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.280+0000 7f3575e00640 1 --2- 172.21.15.28:0/3521433278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 0x7f35700b34f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3568009dc0 tx=0x7f356801c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec2f60b084310f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680b2070 con 0x7f35700cd380 2024-03-21T08:42:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35680a5820 con 0x7f35700cd380 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f356fe00640 1 -- 172.21.15.28:0/3521433278 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680a5bf0 con 0x7f35700cd380 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- 172.21.15.28:0/3521433278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 msgr2=0x7f35700b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 --2- 172.21.15.28:0/3521433278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 0x7f35700b34f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3568009dc0 tx=0x7f356801c870 comp rx=0 tx=0).stop 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- 172.21.15.28:0/3521433278 shutdown_connections 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 --2- 172.21.15.28:0/3521433278 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700cd380 0x7f35700b34f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- 172.21.15.28:0/3521433278 >> 172.21.15.28:0/3521433278 conn(0x7f35700cc230 msgr2=0x7f35700cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- 172.21.15.28:0/3521433278 shutdown_connections 2024-03-21T08:42:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- 172.21.15.28:0/3521433278 wait complete. 2024-03-21T08:42:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 Processor -- start 2024-03-21T08:42:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- start start 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 0x7f357014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35700cd380 0x7f357014b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357015d2a0 con 0x7f357015eb50 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357015d120 con 0x7f35700cd380 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3577e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357015d420 con 0x7f35700b5520 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3575e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3575e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42842/0 (socket says 172.21.15.28:42842) 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3575e00640 1 -- 172.21.15.28:0/3364375520 learned_addr learned my addr 172.21.15.28:0/3364375520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3574a00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35700cd380 0x7f357014b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.284+0000 7f3575400640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 0x7f357014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 -- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 msgr2=0x7f357014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 0x7f357014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 -- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35700cd380 msgr2=0x7f357014b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35700cd380 0x7f357014b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 -- 172.21.15.28:0/3364375520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570151ba0 con 0x7f357015eb50 2024-03-21T08:42:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575400640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 0x7f357014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f356801a4a0 tx=0x7f3568019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680b2050 con 0x7f357015eb50 2024-03-21T08:42:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35680be070 con 0x7f357015eb50 2024-03-21T08:42:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3570151e80 con 0x7f357015eb50 2024-03-21T08:42:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568009450 con 0x7f357015eb50 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3570152440 con 0x7f357015eb50 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357004f410 con 0x7f357015eb50 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f35680ad890 con 0x7f357015eb50 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 0x7f3550069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f35681213b0 con 0x7f357015eb50 2024-03-21T08:42:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.288+0000 7f3575400640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 0x7f3550069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.292+0000 7f3575400640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 0x7f3550069fe0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3564001740 tx=0x7f3564009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.292+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3568016580 con 0x7f357015eb50 2024-03-21T08:42:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.492+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3570072470 con 0x7f357015eb50 2024-03-21T08:42:17.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.492+0000 7f356d600640 1 -- 172.21.15.28:0/3364375520 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7f35680f1a30 con 0x7f357015eb50 2024-03-21T08:42:17.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:17.495 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:42:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 msgr2=0x7f3550069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 0x7f3550069fe0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3564001740 tx=0x7f3564009040 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 msgr2=0x7f357015efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f356801a4a0 tx=0x7f3568019610 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 shutdown_connections 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f357015eb50 0x7f357015efe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f35700cd380 0x7f357014b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3550067b20 0x7f3550069fe0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 --2- 172.21.15.28:0/3364375520 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f35700b5520 0x7f357014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 >> 172.21.15.28:0/3364375520 conn(0x7f35700cc230 msgr2=0x7f3570153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 shutdown_connections 2024-03-21T08:42:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.496+0000 7f3577e00640 1 -- 172.21.15.28:0/3364375520 wait complete. 2024-03-21T08:42:17.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 Processor -- start 2024-03-21T08:42:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 -- start start 2024-03-21T08:42:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ce490 0x7fb0c815e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b4380 0x7fb0c80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c815bf00 con 0x7fb0c815ba70 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c815ebf0 con 0x7fb0c80ce490 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cfe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c815eed0 con 0x7fb0c80b4380 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cd400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ce490 0x7fb0c815e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cd400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ce490 0x7fb0c815e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35762/0 (socket says 172.21.15.28:35762) 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cca00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fb0c815ba70 0x7fb0c80ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42538/0 (socket says 172.21.15.28:42538) 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cd400640 1 -- 172.21.15.28:0/2647366161 learned_addr learned my addr 172.21.15.28:0/2647366161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.656+0000 7fb0cde00640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b4380 0x7fb0c80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cd400640 1 -- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b4380 msgr2=0x7fb0c80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cd400640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b4380 0x7fb0c80b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cd400640 1 -- 172.21.15.28:0/2647366161 >> v1:172.21.15.151:6789/0 conn(0x7fb0c815ba70 legacy=0x7fb0c80ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cd400640 1 -- 172.21.15.28:0/2647366161 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c815f1b0 con 0x7fb0c80ce490 2024-03-21T08:42:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0c7e00640 1 -- 172.21.15.28:0/2647366161 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267822589 0 0) 0x7fb0c815bf00 con 0x7fb0c815ba70 2024-03-21T08:42:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cd400640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ce490 0x7fb0c815e610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b099e1f1cb3ec1f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0c7e00640 1 -- 172.21.15.28:0/2647366161 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0b1070 con 0x7fb0c80ce490 2024-03-21T08:42:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0c7e00640 1 -- 172.21.15.28:0/2647366161 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0ac620 con 0x7fb0c80ce490 2024-03-21T08:42:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0c7e00640 1 -- 172.21.15.28:0/2647366161 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0ac9f0 con 0x7fb0c80ce490 2024-03-21T08:42:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cfe00640 1 -- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ce490 msgr2=0x7fb0c815e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ce490 0x7fb0c815e610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc01c4d0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cfe00640 1 -- 172.21.15.28:0/2647366161 shutdown_connections 2024-03-21T08:42:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b4380 0x7fb0c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.660+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/2647366161 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ce490 0x7fb0c815e610 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- 172.21.15.28:0/2647366161 >> 172.21.15.28:0/2647366161 conn(0x7fb0c80cc230 msgr2=0x7fb0c80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- 172.21.15.28:0/2647366161 shutdown_connections 2024-03-21T08:42:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- 172.21.15.28:0/2647366161 wait complete. 2024-03-21T08:42:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 Processor -- start 2024-03-21T08:42:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- start start 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80ce490 0x7fb0c815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 0x7fb0c814e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8167b30 con 0x7fb0c80b4380 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81679b0 con 0x7fb0c815ba70 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cfe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c8167cb0 con 0x7fb0c80ce490 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42854/0 (socket says 172.21.15.28:42854) 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cd400640 1 -- 172.21.15.28:0/3161436736 learned_addr learned my addr 172.21.15.28:0/3161436736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 0x7fb0c814e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80ce490 0x7fb0c815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 -- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80ce490 msgr2=0x7fb0c815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80ce490 0x7fb0c815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 -- 172.21.15.28:0/3161436736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 msgr2=0x7fb0c8157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 -- 172.21.15.28:0/3161436736 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c8156790 con 0x7fb0c815ba70 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.664+0000 7fb0cde00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 0x7fb0c814e130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000efc0 tx=0x7fb0c000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0018020 con 0x7fb0c815ba70 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0014480 con 0x7fb0c815ba70 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8156a70 con 0x7fb0c815ba70 2024-03-21T08:42:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0014870 con 0x7fb0c815ba70 2024-03-21T08:42:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8157310 con 0x7fb0c815ba70 2024-03-21T08:42:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c8157510 con 0x7fb0c815ba70 2024-03-21T08:42:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb0c001b020 con 0x7fb0c815ba70 2024-03-21T08:42:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 0x7fb0a806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7fb0c0086f60 con 0x7fb0c815ba70 2024-03-21T08:42:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 0x7fb0a806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.668+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 0x7fb0a806a010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.672+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb0c0022030 con 0x7fb0c815ba70 2024-03-21T08:42:17.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.876+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c804f410 con 0x7fb0c815ba70 2024-03-21T08:42:17.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.876+0000 7fb0c5600640 1 -- 172.21.15.28:0/3161436736 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4976 (secure 0 0 0) 0x7fb0c0010040 con 0x7fb0c815ba70 2024-03-21T08:42:17.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:17.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/3880245859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":3880245859},{"type":"v1","addr":"172.21.15.151:6837","nonce":3880245859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/106099657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":106099657},{"type":"v1","addr":"172.21.15.28:6834","nonce":106099657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1150455590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1150455590},{"type":"v1","addr":"172.21.15.28:6835","nonce":1150455590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:41:07.397365+0000","modified":"2024-03-21T08:41:25.469098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.151:6836/4096075958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":4096075958},{"type":"v1","addr":"172.21.15.151:6836","nonce":4096075958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-21T08:42:17.880 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-21T08:42:17.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 msgr2=0x7fb0a806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 0x7fb0a806a010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009870 tx=0x7fb0bc0a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 msgr2=0x7fb0c814e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 0x7fb0c814e130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000efc0 tx=0x7fb0c000c490 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 shutdown_connections 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c815ba70 0x7fb0c814e130 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a8067b50 0x7fb0a806a010 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80ce490 0x7fb0c815ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 --2- 172.21.15.28:0/3161436736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b4380 0x7fb0c8157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 >> 172.21.15.28:0/3161436736 conn(0x7fb0c80cc230 msgr2=0x7fb0c8071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 shutdown_connections 2024-03-21T08:42:17.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:17.880+0000 7fb0cfe00640 1 -- 172.21.15.28:0/3161436736 wait complete. 2024-03-21T08:42:17.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:18.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 Processor -- start 2024-03-21T08:42:18.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 -- start start 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 0x7f5cb006b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 0x7f5cb006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb00d0540 con 0x7f5cb015dd00 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb00cd6e0 con 0x7f5cb00aff70 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb7400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00cd9c0 con 0x7f5cb006eb30 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb4a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5cb015dd00 0x7f5cb00cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39332/0 (socket says 172.21.15.28:39332) 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb4a00640 1 -- 172.21.15.28:0/1918971638 learned_addr learned my addr 172.21.15.28:0/1918971638 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cafe00640 1 --2- 172.21.15.28:0/1918971638 >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 0x7f5cb006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.040+0000 7f5cb5400640 1 --2- 172.21.15.28:0/1918971638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 0x7f5cb006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5caf400640 1 -- 172.21.15.28:0/1918971638 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992651203 0 0) 0x7f5cb00d0540 con 0x7f5cb015dd00 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5caf400640 1 -- 172.21.15.28:0/1918971638 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c98003530 con 0x7f5cb015dd00 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb5400640 1 -- 172.21.15.28:0/1918971638 >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 msgr2=0x7f5cb006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb5400640 1 --2- 172.21.15.28:0/1918971638 >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 0x7f5cb006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb5400640 1 -- 172.21.15.28:0/1918971638 >> v1:172.21.15.28:6789/0 conn(0x7f5cb015dd00 legacy=0x7f5cb00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb5400640 1 -- 172.21.15.28:0/1918971638 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb00cdca0 con 0x7f5cb006eb30 2024-03-21T08:42:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cafe00640 1 --2- 172.21.15.28:0/1918971638 >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 0x7f5cb006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb5400640 1 --2- 172.21.15.28:0/1918971638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 0x7f5cb006b6b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f128154738a0ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5caf400640 1 -- 172.21.15.28:0/1918971638 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca80b2070 con 0x7f5cb006eb30 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5caf400640 1 -- 172.21.15.28:0/1918971638 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca80a5990 con 0x7f5cb006eb30 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5caf400640 1 -- 172.21.15.28:0/1918971638 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca80a5d60 con 0x7f5cb006eb30 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb7400640 1 -- 172.21.15.28:0/1918971638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 msgr2=0x7f5cb006b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.044+0000 7f5cb7400640 1 --2- 172.21.15.28:0/1918971638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 0x7f5cb006b6b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca801c940 comp rx=0 tx=0).stop 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- 172.21.15.28:0/1918971638 shutdown_connections 2024-03-21T08:42:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 --2- 172.21.15.28:0/1918971638 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb006eb30 0x7f5cb006b6b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 --2- 172.21.15.28:0/1918971638 >> v2:172.21.15.151:3300/0 conn(0x7f5cb00aff70 0x7f5cb006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- 172.21.15.28:0/1918971638 >> 172.21.15.28:0/1918971638 conn(0x7f5cb0099a30 msgr2=0x7f5cb0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- 172.21.15.28:0/1918971638 shutdown_connections 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- 172.21.15.28:0/1918971638 wait complete. 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 Processor -- start 2024-03-21T08:42:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- start start 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cb006eb30 0x7f5cb009ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 0x7f5cb0159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0162240 con 0x7f5cb00aff70 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb015e2f0 con 0x7f5cb006eb30 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb01623c0 con 0x7f5cb015dd00 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42862/0 (socket says 172.21.15.28:42862) 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 -- 172.21.15.28:0/2797977979 learned_addr learned my addr 172.21.15.28:0/2797977979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb4a00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cb006eb30 0x7f5cb009ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb5400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 0x7f5cb0159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 -- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 msgr2=0x7f5cb0159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 0x7f5cb0159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 -- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cb006eb30 msgr2=0x7f5cb009ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cb006eb30 0x7f5cb009ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 -- 172.21.15.28:0/2797977979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb009d640 con 0x7f5cb00aff70 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb5400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 0x7f5cb0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cafe00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00ebe0 tx=0x7f5c9c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c017070 con 0x7f5cb00aff70 2024-03-21T08:42:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c00cc30 con 0x7f5cb00aff70 2024-03-21T08:42:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c013470 con 0x7f5cb00aff70 2024-03-21T08:42:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.048+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb015f340 con 0x7f5cb00aff70 2024-03-21T08:42:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb015f900 con 0x7f5cb00aff70 2024-03-21T08:42:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb004f410 con 0x7f5cb00aff70 2024-03-21T08:42:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5c9c01a070 con 0x7f5cb00aff70 2024-03-21T08:42:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cacc00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 0x7f5c90069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f5c9c086310 con 0x7f5cb00aff70 2024-03-21T08:42:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.052+0000 7f5cb4a00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 0x7f5c90069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.056+0000 7f5cb4a00640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 0x7f5c90069fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4007fd0 tx=0x7f5ca4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.056+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5c9c056190 con 0x7f5cb00aff70 2024-03-21T08:42:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.220+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cb00c7230 con 0x7f5cb00aff70 2024-03-21T08:42:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.220+0000 7f5cacc00640 1 -- 172.21.15.28:0/2797977979 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18930 (secure 0 0 0) 0x7f5c9c05a230 con 0x7f5cb00aff70 2024-03-21T08:42:18.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:18.224 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:41:49.139608+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:41:05.747726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-21T08:41:06.377508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-21T08:41:10.058077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1f","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 msgr2=0x7f5c90069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 0x7f5c90069fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4007fd0 tx=0x7f5ca4009040 comp rx=0 tx=0).stop 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 msgr2=0x7f5cb01598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00ebe0 tx=0x7f5c9c00c520 comp rx=0 tx=0).stop 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 shutdown_connections 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5c90067b20 0x7f5c90069fe0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cb015dd00 0x7f5cb0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cb00aff70 0x7f5cb01598f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 --2- 172.21.15.28:0/2797977979 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cb006eb30 0x7f5cb009ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 >> 172.21.15.28:0/2797977979 conn(0x7f5cb0099a30 msgr2=0x7f5cb006ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:18.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 shutdown_connections 2024-03-21T08:42:18.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.224+0000 7f5cb7400640 1 -- 172.21.15.28:0/2797977979 wait complete. 2024-03-21T08:42:18.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-21T08:42:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 Processor -- start 2024-03-21T08:42:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 -- start start 2024-03-21T08:42:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 0x7f2f78168ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7806bd40 con 0x7f2f7815e190 2024-03-21T08:42:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7806c020 con 0x7f2f780d0320 2024-03-21T08:42:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f7e000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7806c300 con 0x7f2f780d0ee0 2024-03-21T08:42:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f6fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2f7815e190 0x7f2f780b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39334/0 (socket says 172.21.15.28:39334) 2024-03-21T08:42:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f6fe00640 1 -- 172.21.15.28:0/3759424420 learned_addr learned my addr 172.21.15.28:0/3759424420 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.384+0000 7f2f77e00640 1 --2- 172.21.15.28:0/3759424420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 0x7f2f78168ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013774737 0 0) 0x7f2f7806bd40 con 0x7f2f7815e190 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003530 con 0x7f2f7815e190 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575163473 0 0) 0x7f2f7806c020 con 0x7f2f780d0320 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f7806bd40 con 0x7f2f780d0320 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f77e00640 1 -- 172.21.15.28:0/3759424420 >> v1:172.21.15.151:6789/0 conn(0x7f2f780d0320 legacy=0x7f2f780d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f77e00640 1 -- 172.21.15.28:0/3759424420 >> v1:172.21.15.28:6789/0 conn(0x7f2f7815e190 legacy=0x7f2f780b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f77e00640 1 -- 172.21.15.28:0/3759424420 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7806c5e0 con 0x7f2f780d0ee0 2024-03-21T08:42:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f77e00640 1 --2- 172.21.15.28:0/3759424420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 0x7f2f78168ab0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009dc0 tx=0x7f2f7001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60237940714e775b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f700b2070 con 0x7f2f780d0ee0 2024-03-21T08:42:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f700a58d0 con 0x7f2f780d0ee0 2024-03-21T08:42:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.388+0000 7f2f76a00640 1 -- 172.21.15.28:0/3759424420 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f700a5ca0 con 0x7f2f780d0ee0 2024-03-21T08:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- 172.21.15.28:0/3759424420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 msgr2=0x7f2f78168ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3759424420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 0x7f2f78168ab0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009dc0 tx=0x7f2f7001c900 comp rx=0 tx=0).stop 2024-03-21T08:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- 172.21.15.28:0/3759424420 shutdown_connections 2024-03-21T08:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3759424420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f780d0ee0 0x7f2f78168ab0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- 172.21.15.28:0/3759424420 >> 172.21.15.28:0/3759424420 conn(0x7f2f78099a30 msgr2=0x7f2f780b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- 172.21.15.28:0/3759424420 shutdown_connections 2024-03-21T08:42:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- 172.21.15.28:0/3759424420 wait complete. 2024-03-21T08:42:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 Processor -- start 2024-03-21T08:42:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- start start 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 0x7f2f7813bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f7815e190 0x7f2f7806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7816bba0 con 0x7f2f780d0320 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7816ba20 con 0x7f2f780d0ee0 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f7e000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7816bd20 con 0x7f2f7815e190 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f6fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 0x7f2f7813bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42872/0 (socket says 172.21.15.28:42872) 2024-03-21T08:42:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 -- 172.21.15.28:0/3357593705 learned_addr learned my addr 172.21.15.28:0/3357593705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77e00640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f7815e190 0x7f2f7806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 -- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f7815e190 msgr2=0x7f2f7806ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f7815e190 0x7f2f7806ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 -- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 msgr2=0x7f2f7813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 0x7f2f7813bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 -- 172.21.15.28:0/3357593705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7814f220 con 0x7f2f780d0320 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f6fe00640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 0x7f2f7813bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.392+0000 7f2f77400640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c4c0 tx=0x7f2f6800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f680052d0 con 0x7f2f780d0320 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68005470 con 0x7f2f780d0320 2024-03-21T08:42:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7814f500 con 0x7f2f780d0320 2024-03-21T08:42:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68012870 con 0x7f2f780d0320 2024-03-21T08:42:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7814fac0 con 0x7f2f780d0320 2024-03-21T08:42:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7804f4f0 con 0x7f2f780d0320 2024-03-21T08:42:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2f68012a10 con 0x7f2f780d0320 2024-03-21T08:42:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 0x7f2f5406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6743+0+0 (secure 0 0 0) 0x7f2f68085e20 con 0x7f2f780d0320 2024-03-21T08:42:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.396+0000 7f2f6fe00640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 0x7f2f5406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.400+0000 7f2f6fe00640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 0x7f2f5406a010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f78153690 tx=0x7f2f60007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.400+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2f68055dd0 con 0x7f2f780d0320 2024-03-21T08:42:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.600+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2f780d0100 con 0x7f2f780d0320 2024-03-21T08:42:18.602 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-03-21T08:42:18.600+0000 7f91dfc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T08:42:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f6f400640 1 -- 172.21.15.28:0/3357593705 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2f68059e70 con 0x7f2f780d0320 2024-03-21T08:42:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-21T08:42:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 msgr2=0x7f2f5406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 0x7f2f5406a010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f78153690 tx=0x7f2f60007d60 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 msgr2=0x7f2f78153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c4c0 tx=0x7f2f6800c980 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 shutdown_connections 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2f54067b50 0x7f2f5406a010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2f7815e190 0x7f2f7806ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2f780d0ee0 0x7f2f7813bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 --2- 172.21.15.28:0/3357593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f780d0320 0x7f2f78153150 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 >> 172.21.15.28:0/3357593705 conn(0x7f2f78099a30 msgr2=0x7f2f78168820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 shutdown_connections 2024-03-21T08:42:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.604+0000 7f2f7e000640 1 -- 172.21.15.28:0/3357593705 wait complete. 2024-03-21T08:42:18.616 INFO:tasks.ceph.mds.d.smithi151.stderr: -4740> 2024-03-21T08:41:11.968+0000 7fe4e5c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:42:18.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-21T08:42:18.665 INFO:tasks.ceph.mds.d.smithi151.stdout:starting mds.d at 2024-03-21T08:42:18.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.768+0000 7f15e8600640 1 Processor -- start 2024-03-21T08:42:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.768+0000 7f15e8600640 1 -- start start 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e00cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 0x7f15e00ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e00b1960 0x7f15e00b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e00cd4e0 con 0x7f15e00b1960 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e00cd9b0 con 0x7f15e00ccb40 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0099e30 con 0x7f15e015bff0 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e00b1960 0x7f15e00b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e00b1960 0x7f15e00b1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42884/0 (socket says 172.21.15.28:42884) 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 -- 172.21.15.28:0/2488569988 learned_addr learned my addr 172.21.15.28:0/2488569988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e5c00640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e00cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e5200640 1 --2- 172.21.15.28:0/2488569988 >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 0x7f15e00ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 -- 172.21.15.28:0/2488569988 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 msgr2=0x7f15e00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e00cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 -- 172.21.15.28:0/2488569988 >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 msgr2=0x7f15e00ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 --2- 172.21.15.28:0/2488569988 >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 0x7f15e00ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e6600640 1 -- 172.21.15.28:0/2488569988 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e009a110 con 0x7f15e00b1960 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e5c00640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e00cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.772+0000 7f15e5200640 1 --2- 172.21.15.28:0/2488569988 >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 0x7f15e00ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e6600640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e00b1df0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f15e00b4720 tx=0x7f15d801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f700b074c3ef75 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15dfe00640 1 -- 172.21.15.28:0/2488569988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d80a5780 con 0x7f15e00b1960 2024-03-21T08:42:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15dfe00640 1 -- 172.21.15.28:0/2488569988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d80a5920 con 0x7f15e00b1960 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15dfe00640 1 -- 172.21.15.28:0/2488569988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d80a5cf0 con 0x7f15e00b1960 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- 172.21.15.28:0/2488569988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 msgr2=0x7f15e00b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e00b1df0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f15e00b4720 tx=0x7f15d801c9f0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- 172.21.15.28:0/2488569988 shutdown_connections 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e00b1df0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 --2- 172.21.15.28:0/2488569988 >> v2:172.21.15.151:3300/0 conn(0x7f15e00ccb40 0x7f15e00ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 --2- 172.21.15.28:0/2488569988 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e00cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- 172.21.15.28:0/2488569988 >> 172.21.15.28:0/2488569988 conn(0x7f15e00b3620 msgr2=0x7f15e00b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- 172.21.15.28:0/2488569988 shutdown_connections 2024-03-21T08:42:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- 172.21.15.28:0/2488569988 wait complete. 2024-03-21T08:42:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 Processor -- start 2024-03-21T08:42:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.776+0000 7f15e8600640 1 -- start start 2024-03-21T08:42:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15e00ccb40 0x7f15e0158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e0150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00cfa90 con 0x7f15e00b1960 2024-03-21T08:42:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00cf910 con 0x7f15e00ccb40 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00cfc10 con 0x7f15e015bff0 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42900/0 (socket says 172.21.15.28:42900) 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 -- 172.21.15.28:0/2215294212 learned_addr learned my addr 172.21.15.28:0/2215294212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5200640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15e00ccb40 0x7f15e0158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e6600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e0150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 -- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 msgr2=0x7f15e0150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e0150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 -- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15e00ccb40 msgr2=0x7f15e0158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15e00ccb40 0x7f15e0158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 -- 172.21.15.28:0/2215294212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e00cb190 con 0x7f15e00b1960 2024-03-21T08:42:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e5c00640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f15d400c820 tx=0x7f15d400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4015070 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d4007a30 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4007e40 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e00cb470 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e015b6f0 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e009faf0 con 0x7f15e00b1960 2024-03-21T08:42:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.780+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f15d4018070 con 0x7f15e00b1960 2024-03-21T08:42:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.784+0000 7f15dd600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 0x7f15c006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.784+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6829+0+0 (secure 0 0 0) 0x7f15d40861d0 con 0x7f15e00b1960 2024-03-21T08:42:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.784+0000 7f15e5200640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 0x7f15c006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.784+0000 7f15e5200640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 0x7f15c006a010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f15e01592b0 tx=0x7f15cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.788+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f15d4056090 con 0x7f15e00b1960 2024-03-21T08:42:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:18.984+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f15e004f4f0 con 0x7f15e00b1960 2024-03-21T08:42:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15dd600640 1 -- 172.21.15.28:0/2215294212 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f15d405a130 con 0x7f15e00b1960 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 msgr2=0x7f15c006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 0x7f15c006a010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f15e01592b0 tx=0x7f15cc008040 comp rx=0 tx=0).stop 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 msgr2=0x7f15e015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f15d400c820 tx=0x7f15d400cce0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 shutdown_connections 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f15c0067b50 0x7f15c006a010 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f15e015bff0 0x7f15e0150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f15e00ccb40 0x7f15e0158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 --2- 172.21.15.28:0/2215294212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e00b1960 0x7f15e015b1b0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.604+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 >> 172.21.15.28:0/2215294212 conn(0x7f15e00b3620 msgr2=0x7f15e0073c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.608+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 shutdown_connections 2024-03-21T08:42:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.608+0000 7f15e8600640 1 -- 172.21.15.28:0/2215294212 wait complete. 2024-03-21T08:42:19.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-21T08:42:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 Processor -- start 2024-03-21T08:42:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 -- start start 2024-03-21T08:42:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 0x7f1aa806b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1aa80cfc00 0x7f1aa80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa80d05a0 con 0x7f1aa815de80 2024-03-21T08:42:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa806ba90 con 0x7f1aa80cfc00 2024-03-21T08:42:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaf200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80cb990 con 0x7f1aa806b170 2024-03-21T08:42:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1aa80cfc00 0x7f1aa80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aade00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 0x7f1aa806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aae800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1aa815de80 0x7f1aa815e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42562/0 (socket says 172.21.15.28:42562) 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aae800640 1 -- 172.21.15.28:0/1837832870 learned_addr learned my addr 172.21.15.28:0/1837832870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1aa80cfc00 0x7f1aa80d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35826/0 (socket says 172.21.15.28:35826) 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 -- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 msgr2=0x7f1aa806b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 0x7f1aa806b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 -- 172.21.15.28:0/1837832870 >> v1:172.21.15.151:6789/0 conn(0x7f1aa815de80 legacy=0x7f1aa815e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aad400640 1 -- 172.21.15.28:0/1837832870 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80cbc70 con 0x7f1aa80cfc00 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aaca00640 1 -- 172.21.15.28:0/1837832870 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082075358 0 0) 0x7f1aa80d05a0 con 0x7f1aa815de80 2024-03-21T08:42:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.776+0000 7f1aade00640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 0x7f1aa806b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:19.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aad400640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa80cfc00 0x7f1aa80d0060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a980097c0 tx=0x7f1a9801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37842d86af81fa45 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaca00640 1 -- 172.21.15.28:0/1837832870 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980a7070 con 0x7f1aa80cfc00 2024-03-21T08:42:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaca00640 1 -- 172.21.15.28:0/1837832870 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a980b0070 con 0x7f1aa80cfc00 2024-03-21T08:42:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaca00640 1 -- 172.21.15.28:0/1837832870 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980ab890 con 0x7f1aa80cfc00 2024-03-21T08:42:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 -- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa80cfc00 msgr2=0x7f1aa80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa80cfc00 0x7f1aa80d0060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a980097c0 tx=0x7f1a9801c470 comp rx=0 tx=0).stop 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 -- 172.21.15.28:0/1837832870 shutdown_connections 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa80cfc00 0x7f1aa80d0060 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 --2- 172.21.15.28:0/1837832870 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa806b170 0x7f1aa806b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 -- 172.21.15.28:0/1837832870 >> 172.21.15.28:0/1837832870 conn(0x7f1aa80b2380 msgr2=0x7f1aa80b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 -- 172.21.15.28:0/1837832870 shutdown_connections 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.780+0000 7f1aaf200640 1 -- 172.21.15.28:0/1837832870 wait complete. 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 Processor -- start 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- start start 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa806b170 0x7f1aa814ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa80cfc00 0x7f1aa80c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa816afb0 con 0x7f1aa815de80 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa815f040 con 0x7f1aa806b170 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa816b130 con 0x7f1aa80cfc00 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42904/0 (socket says 172.21.15.28:42904) 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 -- 172.21.15.28:0/737318511 learned_addr learned my addr 172.21.15.28:0/737318511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aade00640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa806b170 0x7f1aa814ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aad400640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa80cfc00 0x7f1aa80c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 -- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa80cfc00 msgr2=0x7f1aa80c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa80cfc00 0x7f1aa80c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 -- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa806b170 msgr2=0x7f1aa814ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa806b170 0x7f1aa814ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 -- 172.21.15.28:0/737318511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80c65f0 con 0x7f1aa815de80 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aae800640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c7a0 tx=0x7f1aa400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4018070 con 0x7f1aa815de80 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa400a640 con 0x7f1aa815de80 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa80c68d0 con 0x7f1aa815de80 2024-03-21T08:42:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa400aa70 con 0x7f1aa815de80 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa80c6e90 con 0x7f1aa815de80 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.784+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa804f410 con 0x7f1aa815de80 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.788+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1aa401b070 con 0x7f1aa815de80 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.788+0000 7f1aa2000640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 0x7f1a8c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.788+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6829+0+0 (secure 0 0 0) 0x7f1aa40858f0 con 0x7f1aa815de80 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.788+0000 7f1aade00640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 0x7f1a8c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.788+0000 7f1aade00640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 0x7f1a8c069fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b17e0 tx=0x7f1a9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:19.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.792+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1aa4055830 con 0x7f1aa815de80 2024-03-21T08:42:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:19.956+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1aa8067350 con 0x7f1aa815de80 2024-03-21T08:42:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aa2000640 1 -- 172.21.15.28:0/737318511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1aa40598d0 con 0x7f1aa815de80 2024-03-21T08:42:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-03-21T08:42:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 msgr2=0x7f1a8c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 0x7f1a8c069fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b17e0 tx=0x7f1a9c009190 comp rx=0 tx=0).stop 2024-03-21T08:42:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 msgr2=0x7f1aa80ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c7a0 tx=0x7f1aa400cc60 comp rx=0 tx=0).stop 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 shutdown_connections 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa815de80 0x7f1aa80ca260 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1a8c067b20 0x7f1a8c069fe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1aa80cfc00 0x7f1aa80c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 --2- 172.21.15.28:0/737318511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1aa806b170 0x7f1aa814ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 >> 172.21.15.28:0/737318511 conn(0x7f1aa80b2380 msgr2=0x7f1aa8156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.612+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 shutdown_connections 2024-03-21T08:42:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.616+0000 7f1aaf200640 1 -- 172.21.15.28:0/737318511 wait complete. 2024-03-21T08:42:20.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-21T08:42:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 Processor -- start 2024-03-21T08:42:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 -- start start 2024-03-21T08:42:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 0x7f7164071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71640cd2b0 0x7f71640b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716415bfd0 con 0x7f7164071f60 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71640cd8d0 con 0x7f71640cd2b0 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f716bc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716415b230 con 0x7f71640b43e0 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71640cd2b0 0x7f71640b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71640cd2b0 0x7f71640b2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39350/0 (socket says 172.21.15.28:39350) 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 -- 172.21.15.28:0/940496882 learned_addr learned my addr 172.21.15.28:0/940496882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 -- 172.21.15.28:0/940496882 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 msgr2=0x7f7164071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169200640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 0x7f7164071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 0x7f7164071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 -- 172.21.15.28:0/940496882 >> v1:172.21.15.151:6789/0 conn(0x7f7164071f60 legacy=0x7f71640ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169c00640 1 -- 172.21.15.28:0/940496882 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716415b4f0 con 0x7f71640cd2b0 2024-03-21T08:42:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.784+0000 7f7169200640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 0x7f7164071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f7169c00640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640cd2b0 0x7f71640b2a60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f715c009e50 tx=0x7f715c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d4f10fdaec4c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f7163e00640 1 -- 172.21.15.28:0/940496882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c0a5670 con 0x7f71640cd2b0 2024-03-21T08:42:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f7163e00640 1 -- 172.21.15.28:0/940496882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c0a5810 con 0x7f71640cd2b0 2024-03-21T08:42:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f7163e00640 1 -- 172.21.15.28:0/940496882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c0a5be0 con 0x7f71640cd2b0 2024-03-21T08:42:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 -- 172.21.15.28:0/940496882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640cd2b0 msgr2=0x7f71640b2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640cd2b0 0x7f71640b2a60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f715c009e50 tx=0x7f715c01c9f0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 -- 172.21.15.28:0/940496882 shutdown_connections 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640cd2b0 0x7f71640b2a60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 --2- 172.21.15.28:0/940496882 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640b43e0 0x7f7164071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 -- 172.21.15.28:0/940496882 >> 172.21.15.28:0/940496882 conn(0x7f71640674f0 msgr2=0x7f716406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 -- 172.21.15.28:0/940496882 shutdown_connections 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.788+0000 7f716bc00640 1 -- 172.21.15.28:0/940496882 wait complete. 2024-03-21T08:42:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 Processor -- start 2024-03-21T08:42:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- start start 2024-03-21T08:42:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 0x7f7164148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640cd2b0 0x7f7164164f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b37b0 con 0x7f71640b43e0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b49d0 con 0x7f7164071f60 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b3930 con 0x7f71640cd2b0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39356/0 (socket says 172.21.15.28:39356) 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 -- 172.21.15.28:0/2701098257 learned_addr learned my addr 172.21.15.28:0/2701098257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7169c00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640cd2b0 0x7f7164164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7169200640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 0x7f7164148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 -- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640cd2b0 msgr2=0x7f7164164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640cd2b0 0x7f7164164f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 -- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 msgr2=0x7f7164148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 0x7f7164148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 -- 172.21.15.28:0/2701098257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716414adc0 con 0x7f71640b43e0 2024-03-21T08:42:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7169200640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 0x7f7164148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f715be00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f714c00b410 tx=0x7f714c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c015030 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c004020 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c004450 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716414b0a0 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716414b660 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.792+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7164068110 con 0x7f71640b43e0 2024-03-21T08:42:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.796+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f714c018070 con 0x7f71640b43e0 2024-03-21T08:42:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.796+0000 7f7161600640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 0x7f714006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.796+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6391+0+0 (secure 0 0 0) 0x7f714c085c30 con 0x7f71640b43e0 2024-03-21T08:42:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.796+0000 7f7169200640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 0x7f714006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.796+0000 7f7169200640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 0x7f714006a010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7154006fd0 tx=0x7f7154008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.800+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f714c056420 con 0x7f71640b43e0 2024-03-21T08:42:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:20.964+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f716406ec40 con 0x7f71640b43e0 2024-03-21T08:42:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.612+0000 7f7161600640 1 -- 172.21.15.28:0/2701098257 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f714c055d20 con 0x7f71640b43e0 2024-03-21T08:42:21.615 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-03-21T08:42:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 msgr2=0x7f714006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 0x7f714006a010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7154006fd0 tx=0x7f7154008040 comp rx=0 tx=0).stop 2024-03-21T08:42:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 msgr2=0x7f7164164a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f714c00b410 tx=0x7f714c00b8d0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 shutdown_connections 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7140067b50 0x7f714006a010 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7154006fd0 tx=0x7f7154008040 comp rx=0 tx=0).stop 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f71640cd2b0 0x7f7164164f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71640b43e0 0x7f7164164a40 secure :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f714c00b410 tx=0x7f714c00b8d0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 --2- 172.21.15.28:0/2701098257 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7164071f60 0x7f7164148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 >> 172.21.15.28:0/2701098257 conn(0x7f71640674f0 msgr2=0x7f716406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 shutdown_connections 2024-03-21T08:42:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.616+0000 7f716bc00640 1 -- 172.21.15.28:0/2701098257 wait complete. 2024-03-21T08:42:21.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-21T08:42:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 Processor -- start 2024-03-21T08:42:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 -- start start 2024-03-21T08:42:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf00d1600 con 0x7febf0099c20 2024-03-21T08:42:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf00ccab0 con 0x7febf015e050 2024-03-21T08:42:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf6a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf00cc180 con 0x7febf009a080 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febefe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febefe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:53054/0 (socket says 172.21.15.28:53054) 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febf4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54748/0 (socket says 172.21.15.28:54748) 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.808+0000 7febefe00640 1 -- 172.21.15.28:0/1425911829 learned_addr learned my addr 172.21.15.28:0/1425911829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf4a00640 1 -- 172.21.15.28:0/1425911829 >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 msgr2=0x7febf006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf4a00640 1 --2- 172.21.15.28:0/1425911829 >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf4a00640 1 -- 172.21.15.28:0/1425911829 >> v1:172.21.15.151:6789/0 conn(0x7febf0099c20 legacy=0x7febf00ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf4a00640 1 -- 172.21.15.28:0/1425911829 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf00cc460 con 0x7febf009a080 2024-03-21T08:42:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febefe00640 1 --2- 172.21.15.28:0/1425911829 >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf4a00640 1 --2- 172.21.15.28:0/1425911829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febe80097c0 tx=0x7febe801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af552f6e45e8027b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febef400640 1 -- 172.21.15.28:0/1425911829 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe80ab610 con 0x7febf009a080 2024-03-21T08:42:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febef400640 1 -- 172.21.15.28:0/1425911829 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe80b0070 con 0x7febf009a080 2024-03-21T08:42:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febef400640 1 -- 172.21.15.28:0/1425911829 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe80ab9d0 con 0x7febf009a080 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf6a00640 1 -- 172.21.15.28:0/1425911829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 msgr2=0x7febf00d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.812+0000 7febf6a00640 1 --2- 172.21.15.28:0/1425911829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febe80097c0 tx=0x7febe801c410 comp rx=0 tx=0).stop 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- 172.21.15.28:0/1425911829 shutdown_connections 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 --2- 172.21.15.28:0/1425911829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf009a080 0x7febf00d0d60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 --2- 172.21.15.28:0/1425911829 >> v2:172.21.15.151:3300/0 conn(0x7febf015e050 0x7febf006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- 172.21.15.28:0/1425911829 >> 172.21.15.28:0/1425911829 conn(0x7febf00b34f0 msgr2=0x7febf00b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- 172.21.15.28:0/1425911829 shutdown_connections 2024-03-21T08:42:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- 172.21.15.28:0/1425911829 wait complete. 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 Processor -- start 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- start start 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf0099c20 0x7febf014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf009a080 0x7febf0157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00b1140 con 0x7febf015e050 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00b0fc0 con 0x7febf009a080 2024-03-21T08:42:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf6a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf016aaa0 con 0x7febf0099c20 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39366/0 (socket says 172.21.15.28:39366) 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 -- 172.21.15.28:0/992230830 learned_addr learned my addr 172.21.15.28:0/992230830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febefe00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf0099c20 0x7febf014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febe7400640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf009a080 0x7febf0157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 -- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf0099c20 msgr2=0x7febf014b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf0099c20 0x7febf014b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 -- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf009a080 msgr2=0x7febf0157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf009a080 0x7febf0157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 -- 172.21.15.28:0/992230830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf014bcf0 con 0x7febf015e050 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.816+0000 7febf4a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7febe801a040 tx=0x7febe801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe80abe10 con 0x7febf015e050 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe80bf410 con 0x7febf015e050 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe80bf800 con 0x7febf015e050 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf0158760 con 0x7febf015e050 2024-03-21T08:42:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf0158d20 con 0x7febf015e050 2024-03-21T08:42:21.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0068110 con 0x7febf015e050 2024-03-21T08:42:21.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7febe80bd020 con 0x7febf015e050 2024-03-21T08:42:21.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 0x7febcc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:21.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5853+0+0 (secure 0 0 0) 0x7febe8120260 con 0x7febf015e050 2024-03-21T08:42:21.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febefe00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 0x7febcc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:21.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.820+0000 7febefe00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 0x7febcc069fe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7febe0009c90 tx=0x7febe000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:21.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.824+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7febf014bcf0 con 0x7febf015e050 2024-03-21T08:42:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:21.988+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7febf00cd2c0 con 0x7febf015e050 2024-03-21T08:42:22.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.632+0000 7febecc00640 1 -- 172.21.15.28:0/992230830 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7febe80a8050 con 0x7febf015e050 2024-03-21T08:42:22.633 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-03-21T08:42:22.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 msgr2=0x7febcc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 0x7febcc069fe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7febe0009c90 tx=0x7febe000b040 comp rx=0 tx=0).stop 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 msgr2=0x7febf0158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7febe801a040 tx=0x7febe801ac50 comp rx=0 tx=0).stop 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 shutdown_connections 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febf015e050 0x7febf0158130 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7febf009a080 0x7febf0157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7febcc067b20 0x7febcc069fe0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 --2- 172.21.15.28:0/992230830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7febf0099c20 0x7febf014b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 >> 172.21.15.28:0/992230830 conn(0x7febf00b34f0 msgr2=0x7febf00b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:22.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 shutdown_connections 2024-03-21T08:42:22.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.636+0000 7febf6a00640 1 -- 172.21.15.28:0/992230830 wait complete. 2024-03-21T08:42:22.662 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-21T08:42:22.662 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-21T08:42:22.662 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-21T08:42:22.662 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:22.663 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-21T08:42:22.663 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-21T08:42:22.663 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-21T08:42:22.663 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:22.663 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-21T08:42:22.663 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-21T08:42:22.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-21T08:42:22.663 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:22.664 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-21T08:42:22.664 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-21T08:42:22.664 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-21T08:42:22.664 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:22.677 INFO:tasks.ceph.mds.d:Stopped 2024-03-21T08:42:22.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-21T08:42:22.678 INFO:tasks.ceph.mds.b:Stopped 2024-03-21T08:42:22.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-21T08:42:22.680 INFO:tasks.ceph.mds.c:Stopped 2024-03-21T08:42:22.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-21T08:42:22.681 INFO:tasks.ceph.mds.a:Stopped 2024-03-21T08:42:22.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-21T08:42:22.725 INFO:tasks.ceph.mds.d:Started 2024-03-21T08:42:22.726 DEBUG:teuthology.parallel:result is None 2024-03-21T08:42:22.727 INFO:tasks.ceph.mds.b:Started 2024-03-21T08:42:22.727 DEBUG:teuthology.parallel:result is None 2024-03-21T08:42:22.730 INFO:tasks.ceph.mds.c:Started 2024-03-21T08:42:22.731 DEBUG:teuthology.parallel:result is None 2024-03-21T08:42:22.732 INFO:tasks.ceph.mds.a:Started 2024-03-21T08:42:22.732 DEBUG:teuthology.parallel:result is None 2024-03-21T08:42:22.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-21T08:42:22.814 INFO:tasks.ceph.mds.b.smithi151.stdout:starting mds.b at 2024-03-21T08:42:22.814 INFO:tasks.ceph.mds.d.smithi151.stdout:starting mds.d at 2024-03-21T08:42:22.819 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-03-21T08:42:22.820 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-03-21T08:42:22.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-21T08:42:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 Processor -- start 2024-03-21T08:42:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 -- start start 2024-03-21T08:42:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea815dba0 0x7f7ea80cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea80ce3d0 con 0x7f7ea8071c90 2024-03-21T08:42:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea80ce6b0 con 0x7f7ea80d0140 2024-03-21T08:42:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7eacc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea80cc340 con 0x7f7ea815dba0 2024-03-21T08:42:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea815dba0 0x7f7ea80cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea815dba0 0x7f7ea80cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39416/0 (socket says 172.21.15.28:39416) 2024-03-21T08:42:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 -- 172.21.15.28:0/787743169 learned_addr learned my addr 172.21.15.28:0/787743169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 -- 172.21.15.28:0/787743169 >> v1:172.21.15.151:6789/0 conn(0x7f7ea80d0140 legacy=0x7f7ea80cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 -- 172.21.15.28:0/787743169 >> v1:172.21.15.28:6789/0 conn(0x7f7ea8071c90 legacy=0x7f7ea80cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.880+0000 7f7ea7400640 1 -- 172.21.15.28:0/787743169 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80ce990 con 0x7f7ea815dba0 2024-03-21T08:42:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7ea7400640 1 --2- 172.21.15.28:0/787743169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80cddf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009e50 tx=0x7f7e9801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67d3d92b6b87bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7ea6a00640 1 -- 172.21.15.28:0/787743169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e980a5700 con 0x7f7ea815dba0 2024-03-21T08:42:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7ea6a00640 1 -- 172.21.15.28:0/787743169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e980a58a0 con 0x7f7ea815dba0 2024-03-21T08:42:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7ea6a00640 1 -- 172.21.15.28:0/787743169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e980a5c70 con 0x7f7ea815dba0 2024-03-21T08:42:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 -- 172.21.15.28:0/787743169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 msgr2=0x7f7ea80cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 --2- 172.21.15.28:0/787743169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80cddf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009e50 tx=0x7f7e9801ca30 comp rx=0 tx=0).stop 2024-03-21T08:42:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 -- 172.21.15.28:0/787743169 shutdown_connections 2024-03-21T08:42:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 --2- 172.21.15.28:0/787743169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80cddf0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 -- 172.21.15.28:0/787743169 >> 172.21.15.28:0/787743169 conn(0x7f7ea806fa90 msgr2=0x7f7ea806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 -- 172.21.15.28:0/787743169 shutdown_connections 2024-03-21T08:42:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.884+0000 7f7eacc00640 1 -- 172.21.15.28:0/787743169 wait complete. 2024-03-21T08:42:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 Processor -- start 2024-03-21T08:42:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- start start 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ea8071c90 0x7f7ea8160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 0x7f7ea80c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80cd870 con 0x7f7ea815dba0 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80cd6f0 con 0x7f7ea80d0140 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80cd9f0 con 0x7f7ea8071c90 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39428/0 (socket says 172.21.15.28:39428) 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 -- 172.21.15.28:0/3666877661 learned_addr learned my addr 172.21.15.28:0/3666877661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7e9ea00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 0x7f7ea80c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7400640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ea8071c90 0x7f7ea8160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 -- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ea8071c90 msgr2=0x7f7ea8160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ea8071c90 0x7f7ea8160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 -- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 msgr2=0x7f7ea80c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 0x7f7ea80c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 -- 172.21.15.28:0/3666877661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80ca700 con 0x7f7ea815dba0 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7e9ea00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 0x7f7ea80c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7ea7e00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c430 tx=0x7f7ea000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0005220 con 0x7f7ea815dba0 2024-03-21T08:42:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea00053c0 con 0x7f7ea815dba0 2024-03-21T08:42:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0007270 con 0x7f7ea815dba0 2024-03-21T08:42:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea80ca9e0 con 0x7f7ea815dba0 2024-03-21T08:42:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea8150b50 con 0x7f7ea815dba0 2024-03-21T08:42:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.888+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea8150d50 con 0x7f7ea815dba0 2024-03-21T08:42:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.892+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7ea0019020 con 0x7f7ea815dba0 2024-03-21T08:42:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.892+0000 7f7e9fe00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 0x7f7e84069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.892+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f7ea00850a0 con 0x7f7ea815dba0 2024-03-21T08:42:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.892+0000 7f7ea7400640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 0x7f7e84069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.892+0000 7f7ea7400640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 0x7f7e84069fe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e980ae000 tx=0x7f7e980a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:22.896+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ea8150d50 con 0x7f7ea815dba0 2024-03-21T08:42:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.072+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7ea804f410 con 0x7f7ea815dba0 2024-03-21T08:42:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.644+0000 7f7e9fe00640 1 -- 172.21.15.28:0/3666877661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7ea0055c20 con 0x7f7ea815dba0 2024-03-21T08:42:23.646 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-03-21T08:42:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 msgr2=0x7f7e84069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 0x7f7e84069fe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e980ae000 tx=0x7f7e980a7070 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 msgr2=0x7f7ea80c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000c430 tx=0x7f7ea000c8f0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 shutdown_connections 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea815dba0 0x7f7ea80c9ec0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ea80d0140 0x7f7ea80c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7e84067b20 0x7f7e84069fe0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 --2- 172.21.15.28:0/3666877661 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7ea8071c90 0x7f7ea8160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 >> 172.21.15.28:0/3666877661 conn(0x7f7ea806fa90 msgr2=0x7f7ea815f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 shutdown_connections 2024-03-21T08:42:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.648+0000 7f7eacc00640 1 -- 172.21.15.28:0/3666877661 wait complete. 2024-03-21T08:42:23.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-21T08:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.816+0000 7f94a0400640 1 Processor -- start 2024-03-21T08:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.816+0000 7f94a0400640 1 -- start start 2024-03-21T08:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 0x7f94980cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94980b34f0 0x7f94980b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 0x7f9498071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980ce6e0 con 0x7f94980b34f0 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949815bf00 con 0x7f94980af060 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f94a0400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980b3980 con 0x7f94980b5350 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94980b34f0 0x7f94980b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94980b34f0 0x7f94980b4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39440/0 (socket says 172.21.15.28:39440) 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 -- 172.21.15.28:0/497498697 learned_addr learned my addr 172.21.15.28:0/497498697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949da00640 1 --2- 172.21.15.28:0/497498697 >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 0x7f94980cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949e400640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 0x7f9498071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 -- 172.21.15.28:0/497498697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 msgr2=0x7f9498071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 0x7f9498071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 -- 172.21.15.28:0/497498697 >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 msgr2=0x7f94980cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 --2- 172.21.15.28:0/497498697 >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 0x7f94980cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949d000640 1 -- 172.21.15.28:0/497498697 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980cdad0 con 0x7f94980b34f0 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949da00640 1 --2- 172.21.15.28:0/497498697 >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 0x7f94980cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.820+0000 7f949e400640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 0x7f9498071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f949d000640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b34f0 0x7f94980b4e10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9498073140 tx=0x7f948401c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb22e1d991b1e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f9497e00640 1 -- 172.21.15.28:0/497498697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840b0070 con 0x7f94980b34f0 2024-03-21T08:42:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f9497e00640 1 -- 172.21.15.28:0/497498697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94840ab600 con 0x7f94980b34f0 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f9497e00640 1 -- 172.21.15.28:0/497498697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840ab9d0 con 0x7f94980b34f0 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- 172.21.15.28:0/497498697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b34f0 msgr2=0x7f94980b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b34f0 0x7f94980b4e10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9498073140 tx=0x7f948401c1d0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- 172.21.15.28:0/497498697 shutdown_connections 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94980b5350 0x7f9498071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- 172.21.15.28:0/497498697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b34f0 0x7f94980b4e10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- 172.21.15.28:0/497498697 >> v2:172.21.15.151:3300/0 conn(0x7f94980af060 0x7f94980cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- 172.21.15.28:0/497498697 >> 172.21.15.28:0/497498697 conn(0x7f94980cc230 msgr2=0x7f94980cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- 172.21.15.28:0/497498697 shutdown_connections 2024-03-21T08:42:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- 172.21.15.28:0/497498697 wait complete. 2024-03-21T08:42:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 Processor -- start 2024-03-21T08:42:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- start start 2024-03-21T08:42:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 0x7f949815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9498153c70 0x7f94980acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980b5730 con 0x7f94980b5350 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949815c200 con 0x7f94980b34f0 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.824+0000 7f94a0400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949815d9c0 con 0x7f9498153c70 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949da00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 0x7f949815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949da00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 0x7f949815d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:53116/0 (socket says 172.21.15.28:53116) 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949da00640 1 -- 172.21.15.28:0/1416952025 learned_addr learned my addr 172.21.15.28:0/1416952025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39454/0 (socket says 172.21.15.28:39454) 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949e400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9498153c70 0x7f94980acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 -- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9498153c70 msgr2=0x7f94980acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9498153c70 0x7f94980acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 -- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 msgr2=0x7f949815d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 0x7f949815d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 -- 172.21.15.28:0/1416952025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949815dbf0 con 0x7f94980b5350 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f949d000640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f94840ab000 tx=0x7f9484018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840b0040 con 0x7f94980b5350 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94840b4440 con 0x7f94980b5350 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94840b4830 con 0x7f94980b5350 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949815ded0 con 0x7f94980b5350 2024-03-21T08:42:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949815e490 con 0x7f94980b5350 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949804f410 con 0x7f94980b5350 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.828+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f94840bb020 con 0x7f94980b5350 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.832+0000 7f9495600640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 0x7f9478069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.832+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f948411f1b0 con 0x7f94980b5350 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.832+0000 7f949da00640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 0x7f9478069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.832+0000 7f949da00640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 0x7f9478069fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94980b34c0 tx=0x7f948c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:23.832+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f94840c2020 con 0x7f94980b5350 2024-03-21T08:42:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.052+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f94980ccd50 con 0x7f94980b5350 2024-03-21T08:42:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.052+0000 7f9495600640 1 -- 172.21.15.28:0/1416952025 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f9484016110 con 0x7f94980b5350 2024-03-21T08:42:24.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtl+trHARAAVeUAY/NXkRXJdfWXcNEaCg==", 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.056 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlQySxDRAA2TpyxtlBbuQn0qvoIOJIQQ==", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlDRkWCRAAilhyLYX1c0vULfdMLzWWAg==", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:42:24.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlnDLEFBAA6RwfJD0dSbvyuYmxKzSszA==", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBr8vtlK0S6NRAAKeHWfA4L/uoAhGbJeQ1p2w==", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBu8vtlGs2gOhAAFHS1xJ/R1i2EgFOJRLnTlw==", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBy8vtl5TVhBBAAxqdX889ZTIy+UrQxJ5f2fg==", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB18vtlpyuwCRAAUZq5F4NJnBLa2+w5o/JKyA==", 2024-03-21T08:42:24.060 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB78vtlOZsrExAAH0sYZ+QeHoo4NMVzEkGg3Q==", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB+8vtld4LJGRAAl/n0wJSkXXTQBqPgxXyEHQ==", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCB8vtlzRpuHxAAeRNXpyRn0AxBrPcyKN9WHw==", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCE8vtlicn7JBAAv8yEBq2UMrr+rKEVspGVRQ==", 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.063 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw==", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtl4oOHIhAARXBoaT5pt4yfDvlIpcpweg==", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:42:24.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlouglHxAAs+T0GLqKcfGvetFHJxp01Q==", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtljj+9KBAAA3rqu+R19oGWN/hi0PVTdQ==", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtl+FA5IhAAgiEdsNSEbXexGkarmJyHJg==", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:42:24.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtljVyzJBAABYLU3AFhSsthTOQT0fAgDQ==", 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtlqmazJBAAUd5PU87givi1qg75BZBJqw==", 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-21T08:42:24.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtlBG+zJBAAhQgAo4+s/GHhCZe2yx6koQ==", 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.071 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtlJXezJBAAz60cTBB8porMLqq615bRZw==", 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-21T08:42:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtlWoCzJBAAH9xzFDdsQCCkqFKSu1VVlQ==", 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCK8vtl6IizJBAAq8ObFAXcKxGRIAA34q3FCA==", 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtlA+USORAA7ZqV/pA9y1+4XBehbbumog==", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtl+il8NRAAUlToWvhBAZuAqKX8f58AOg==", 2024-03-21T08:42:24.074 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-21T08:42:24.075 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 msgr2=0x7f9478069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 0x7f9478069fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94980b34c0 tx=0x7f948c008040 comp rx=0 tx=0).stop 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 msgr2=0x7f949813e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f94840ab000 tx=0x7f9484018f70 comp rx=0 tx=0).stop 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 shutdown_connections 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9478067b20 0x7f9478069fe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9498153c70 0x7f94980acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94980b5350 0x7f949813e080 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 --2- 172.21.15.28:0/1416952025 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94980b34f0 0x7f949815d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 >> 172.21.15.28:0/1416952025 conn(0x7f94980cc230 msgr2=0x7f949815eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 shutdown_connections 2024-03-21T08:42:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.056+0000 7f94a0400640 1 -- 172.21.15.28:0/1416952025 wait complete. 2024-03-21T08:42:24.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-21T08:42:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 Processor -- start 2024-03-21T08:42:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 -- start start 2024-03-21T08:42:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba815bff0 0x7faba80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faba80b2970 0x7faba80cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba80cd9b0 con 0x7faba80ccb40 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba8099e30 con 0x7faba80b2970 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabafc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba809a110 con 0x7faba815bff0 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faba80b2970 0x7faba80cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faba80b2970 0x7faba80cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39466/0 (socket says 172.21.15.28:39466) 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 -- 172.21.15.28:0/3223885421 learned_addr learned my addr 172.21.15.28:0/3223885421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabad200640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba815bff0 0x7faba80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 -- 172.21.15.28:0/3223885421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba815bff0 msgr2=0x7faba80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba815bff0 0x7faba80cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 -- 172.21.15.28:0/3223885421 >> v1:172.21.15.151:6789/0 conn(0x7faba80ccb40 legacy=0x7faba80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.220+0000 7fabadc00640 1 -- 172.21.15.28:0/3223885421 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba809a3f0 con 0x7faba80b2970 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabadc00640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba80b2970 0x7faba80cf700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7faba0009e50 tx=0x7faba001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8926060c1a183bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7faba7400640 1 -- 172.21.15.28:0/3223885421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00a5810 con 0x7faba80b2970 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7faba7400640 1 -- 172.21.15.28:0/3223885421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00a59b0 con 0x7faba80b2970 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7faba7400640 1 -- 172.21.15.28:0/3223885421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00a5d80 con 0x7faba80b2970 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 -- 172.21.15.28:0/3223885421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba80b2970 msgr2=0x7faba80cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba80b2970 0x7faba80cf700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7faba0009e50 tx=0x7faba001ca30 comp rx=0 tx=0).stop 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 -- 172.21.15.28:0/3223885421 shutdown_connections 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba80b2970 0x7faba80cf700 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 --2- 172.21.15.28:0/3223885421 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba815bff0 0x7faba80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 -- 172.21.15.28:0/3223885421 >> 172.21.15.28:0/3223885421 conn(0x7faba80b3620 msgr2=0x7faba80b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 -- 172.21.15.28:0/3223885421 shutdown_connections 2024-03-21T08:42:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.224+0000 7fabafc00640 1 -- 172.21.15.28:0/3223885421 wait complete. 2024-03-21T08:42:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 Processor -- start 2024-03-21T08:42:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- start start 2024-03-21T08:42:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faba80b2970 0x7faba80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 0x7faba8154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba80d07e0 con 0x7faba815bff0 2024-03-21T08:42:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba80d0660 con 0x7faba80b2970 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba80d0960 con 0x7faba80ccb40 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39470/0 (socket says 172.21.15.28:39470) 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 -- 172.21.15.28:0/4172382057 learned_addr learned my addr 172.21.15.28:0/4172382057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabad200640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faba80b2970 0x7faba80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7faba7e00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 0x7faba8154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 -- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 msgr2=0x7faba8154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 0x7faba8154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 -- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faba80b2970 msgr2=0x7faba80afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faba80b2970 0x7faba80afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 -- 172.21.15.28:0/4172382057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba81548d0 con 0x7faba815bff0 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7faba7e00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 0x7faba8154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabadc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7faba001a660 tx=0x7faba00197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00bc020 con 0x7faba815bff0 2024-03-21T08:42:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00b8880 con 0x7faba815bff0 2024-03-21T08:42:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0009510 con 0x7faba815bff0 2024-03-21T08:42:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba8156310 con 0x7faba815bff0 2024-03-21T08:42:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.228+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba81568d0 con 0x7faba815bff0 2024-03-21T08:42:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba809faf0 con 0x7faba815bff0 2024-03-21T08:42:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faba00bf020 con 0x7faba815bff0 2024-03-21T08:42:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7faba4c00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 0x7fab8806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7faba0120f10 con 0x7faba815bff0 2024-03-21T08:42:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7fabad200640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 0x7fab8806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.232+0000 7fabad200640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 0x7fab8806a010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009b60 tx=0x7fab9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.236+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faba00c6030 con 0x7faba815bff0 2024-03-21T08:42:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.455+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7faba809a8e0 con 0x7faba815bff0 2024-03-21T08:42:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.455+0000 7faba4c00640 1 -- 172.21.15.28:0/4172382057 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7faba0016620 con 0x7faba815bff0 2024-03-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 msgr2=0x7fab8806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 0x7fab8806a010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009b60 tx=0x7fab9c009190 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 msgr2=0x7faba8157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7faba001a660 tx=0x7faba00197d0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 shutdown_connections 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faba815bff0 0x7faba8157690 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fab88067b50 0x7fab8806a010 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faba80ccb40 0x7faba8154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 --2- 172.21.15.28:0/4172382057 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faba80b2970 0x7faba80afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 >> 172.21.15.28:0/4172382057 conn(0x7faba80b3620 msgr2=0x7faba809ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 shutdown_connections 2024-03-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.459+0000 7fabafc00640 1 -- 172.21.15.28:0/4172382057 wait complete. 2024-03-21T08:42:24.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-21T08:42:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 Processor -- start 2024-03-21T08:42:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 -- start start 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad415bca0 0x7fbad415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbad40cfa70 0x7fbad40cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad415d050 con 0x7fbad40cf600 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad406b2f0 con 0x7fbad40cfa70 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad406b5d0 con 0x7fbad415bca0 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbad40cfa70 0x7fbad40cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbad40cfa70 0x7fbad40cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39482/0 (socket says 172.21.15.28:39482) 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 -- 172.21.15.28:0/2056600919 learned_addr learned my addr 172.21.15.28:0/2056600919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad3e00640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad415bca0 0x7fbad415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 -- 172.21.15.28:0/2056600919 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad415bca0 msgr2=0x7fbad415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad415bca0 0x7fbad415cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 -- 172.21.15.28:0/2056600919 >> v1:172.21.15.151:6789/0 conn(0x7fbad40cf600 legacy=0x7fbad40cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.619+0000 7fbad9000640 1 -- 172.21.15.28:0/2056600919 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad406b8b0 con 0x7fbad40cfa70 2024-03-21T08:42:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9000640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cfa70 0x7fbad40cff20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009e50 tx=0x7fbacc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162cc91b53db8cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad2a00640 1 -- 172.21.15.28:0/2056600919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0a5660 con 0x7fbad40cfa70 2024-03-21T08:42:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad2a00640 1 -- 172.21.15.28:0/2056600919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc0a5800 con 0x7fbad40cfa70 2024-03-21T08:42:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad2a00640 1 -- 172.21.15.28:0/2056600919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0a5bd0 con 0x7fbad40cfa70 2024-03-21T08:42:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 -- 172.21.15.28:0/2056600919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cfa70 msgr2=0x7fbad40cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cfa70 0x7fbad40cff20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009e50 tx=0x7fbacc01c9a0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 -- 172.21.15.28:0/2056600919 shutdown_connections 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cfa70 0x7fbad40cff20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 --2- 172.21.15.28:0/2056600919 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad415bca0 0x7fbad415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 -- 172.21.15.28:0/2056600919 >> 172.21.15.28:0/2056600919 conn(0x7fbad40b2210 msgr2=0x7fbad40b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 -- 172.21.15.28:0/2056600919 shutdown_connections 2024-03-21T08:42:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.623+0000 7fbad9a00640 1 -- 172.21.15.28:0/2056600919 wait complete. 2024-03-21T08:42:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 Processor -- start 2024-03-21T08:42:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- start start 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad40cfa70 0x7fbad41f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad40adbb0 0x7fbad40b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40cf1d0 con 0x7fbad40cf600 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40cf050 con 0x7fbad40cfa70 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41686d0 con 0x7fbad40adbb0 2024-03-21T08:42:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39498/0 (socket says 172.21.15.28:39498) 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 -- 172.21.15.28:0/3256413686 learned_addr learned my addr 172.21.15.28:0/3256413686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9000640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad40adbb0 0x7fbad40b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3400640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad40cfa70 0x7fbad41f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 -- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad40adbb0 msgr2=0x7fbad40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad40adbb0 0x7fbad40b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 -- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad40cfa70 msgr2=0x7fbad41f6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad40cfa70 0x7fbad41f6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 -- 172.21.15.28:0/3256413686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad41f9e50 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad3e00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbac400e930 tx=0x7fbac400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4017070 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac400ca00 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac400cdb0 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad40b3ed0 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.627+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad40b5f00 con 0x7fbad40cf600 2024-03-21T08:42:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad404f410 con 0x7fbad40cf600 2024-03-21T08:42:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbac401a070 con 0x7fbad40cf600 2024-03-21T08:42:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbacbe00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 0x7fbab4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbac4085ce0 con 0x7fbad40cf600 2024-03-21T08:42:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbad3400640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 0x7fbab4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.631+0000 7fbad3400640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 0x7fbab4069fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbad40c4120 tx=0x7fbac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.635+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbad41f9e50 con 0x7fbad40cf600 2024-03-21T08:42:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.859+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fbad40b6140 con 0x7fbad40cf600 2024-03-21T08:42:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbacbe00640 1 -- 172.21.15.28:0/3256413686 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbac4056310 con 0x7fbad40cf600 2024-03-21T08:42:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 msgr2=0x7fbab4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 0x7fbab4069fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbad40c4120 tx=0x7fbac0009040 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 msgr2=0x7fbad41f9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbac400e930 tx=0x7fbac400edf0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 shutdown_connections 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbab4067b20 0x7fbab4069fe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbad40adbb0 0x7fbad40b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbad40cfa70 0x7fbad41f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 --2- 172.21.15.28:0/3256413686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad40cf600 0x7fbad41f9910 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 >> 172.21.15.28:0/3256413686 conn(0x7fbad40b2210 msgr2=0x7fbad406ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 shutdown_connections 2024-03-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:24.863+0000 7fbad9a00640 1 -- 172.21.15.28:0/3256413686 wait complete. 2024-03-21T08:42:24.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-21T08:42:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.027+0000 7fbd81200640 1 Processor -- start 2024-03-21T08:42:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.027+0000 7fbd81200640 1 -- start start 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd81200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbd7c072190 0x7fbd7c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd81200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c15bf90 con 0x7fbd7c0b4420 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd81200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7c15b1d0 con 0x7fbd7c0b4800 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd81200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7c15b4b0 con 0x7fbd7c072190 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd79e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd7c0b4420 0x7fbd7c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49784/0 (socket says 172.21.15.28:49784) 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd79e00640 1 -- 172.21.15.28:0/3982182454 learned_addr learned my addr 172.21.15.28:0/3982182454 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd7a800640 1 --2- 172.21.15.28:0/3982182454 >> v2:172.21.15.28:3300/0 conn(0x7fbd7c072190 0x7fbd7c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd7a800640 1 -- 172.21.15.28:0/3982182454 >> v1:172.21.15.151:6789/0 conn(0x7fbd7c0b4800 legacy=0x7fbd7c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd79400640 1 -- 172.21.15.28:0/3982182454 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537144635 0 0) 0x7fbd7c15bf90 con 0x7fbd7c0b4420 2024-03-21T08:42:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd7a800640 1 -- 172.21.15.28:0/3982182454 >> v1:172.21.15.28:6789/0 conn(0x7fbd7c0b4420 legacy=0x7fbd7c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.031+0000 7fbd7a800640 1 -- 172.21.15.28:0/3982182454 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c15b790 con 0x7fbd7c072190 2024-03-21T08:42:25.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd7a800640 1 --2- 172.21.15.28:0/3982182454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c072190 0x7fbd7c0cd300 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009d90 tx=0x7fbd7401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edb347237343250 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd79400640 1 -- 172.21.15.28:0/3982182454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740b2070 con 0x7fbd7c072190 2024-03-21T08:42:25.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd79400640 1 -- 172.21.15.28:0/3982182454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd740a59a0 con 0x7fbd7c072190 2024-03-21T08:42:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd79400640 1 -- 172.21.15.28:0/3982182454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740a5d70 con 0x7fbd7c072190 2024-03-21T08:42:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- 172.21.15.28:0/3982182454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c072190 msgr2=0x7fbd7c0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 --2- 172.21.15.28:0/3982182454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c072190 0x7fbd7c0cd300 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009d90 tx=0x7fbd7401c940 comp rx=0 tx=0).stop 2024-03-21T08:42:25.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- 172.21.15.28:0/3982182454 shutdown_connections 2024-03-21T08:42:25.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 --2- 172.21.15.28:0/3982182454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c072190 0x7fbd7c0cd300 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- 172.21.15.28:0/3982182454 >> 172.21.15.28:0/3982182454 conn(0x7fbd7c0674f0 msgr2=0x7fbd7c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:25.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- 172.21.15.28:0/3982182454 shutdown_connections 2024-03-21T08:42:25.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- 172.21.15.28:0/3982182454 wait complete. 2024-03-21T08:42:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 Processor -- start 2024-03-21T08:42:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 -- start start 2024-03-21T08:42:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.035+0000 7fbd81200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 0x7fbd7c144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 0x7fbd7c155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c168140 con 0x7fbd7c145140 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c167fc0 con 0x7fbd7c0b4420 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c1682c0 con 0x7fbd7c072190 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39524/0 (socket says 172.21.15.28:39524) 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 -- 172.21.15.28:0/272331462 learned_addr learned my addr 172.21.15.28:0/272331462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd79e00640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 0x7fbd7c144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd71400640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 0x7fbd7c155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 -- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 msgr2=0x7fbd7c144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 0x7fbd7c144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 -- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 msgr2=0x7fbd7c155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 0x7fbd7c155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 -- 172.21.15.28:0/272331462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c06bec0 con 0x7fbd7c145140 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd79e00640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 0x7fbd7c144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd71400640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 0x7fbd7c155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd7a800640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbd7401a570 tx=0x7fbd740196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740b2050 con 0x7fbd7c145140 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd740be070 con 0x7fbd7c145140 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740adad0 con 0x7fbd7c145140 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7c150d80 con 0x7fbd7c145140 2024-03-21T08:42:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c151340 con 0x7fbd7c145140 2024-03-21T08:42:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7c073430 con 0x7fbd7c145140 2024-03-21T08:42:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.039+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbd740adc70 con 0x7fbd7c145140 2024-03-21T08:42:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.043+0000 7fbd72a00640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 0x7fbd5806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.043+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbd74120b00 con 0x7fbd7c145140 2024-03-21T08:42:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.043+0000 7fbd79e00640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 0x7fbd5806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.043+0000 7fbd79e00640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 0x7fbd5806a010 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c000f70 tx=0x7fbd6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.047+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbd740c7020 con 0x7fbd7c145140 2024-03-21T08:42:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.263+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fbd7c15a870 con 0x7fbd7c145140 2024-03-21T08:42:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd72a00640 1 -- 172.21.15.28:0/272331462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbd74016560 con 0x7fbd7c145140 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 msgr2=0x7fbd5806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 0x7fbd5806a010 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c000f70 tx=0x7fbd6c009040 comp rx=0 tx=0).stop 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 msgr2=0x7fbd7c145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbd7401a570 tx=0x7fbd740196e0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 shutdown_connections 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd7c145140 0x7fbd7c145590 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbd7c0b4420 0x7fbd7c155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbd58067b50 0x7fbd5806a010 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 --2- 172.21.15.28:0/272331462 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbd7c072190 0x7fbd7c144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 >> 172.21.15.28:0/272331462 conn(0x7fbd7c0674f0 msgr2=0x7fbd7c15d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 shutdown_connections 2024-03-21T08:42:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.267+0000 7fbd81200640 1 -- 172.21.15.28:0/272331462 wait complete. 2024-03-21T08:42:25.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-21T08:42:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.431+0000 7f904ca00640 1 Processor -- start 2024-03-21T08:42:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.431+0000 7f904ca00640 1 -- start start 2024-03-21T08:42:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.431+0000 7f904ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9048072190 0x7f90480cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.431+0000 7f904ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904815bf90 con 0x7f90480b4420 2024-03-21T08:42:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.431+0000 7f904ca00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904815b1d0 con 0x7f90480b4800 2024-03-21T08:42:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f904ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904815b4b0 con 0x7f9048072190 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90480b4420 0x7f9048071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49790/0 (socket says 172.21.15.28:49790) 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045e00640 1 -- 172.21.15.28:0/3007731005 learned_addr learned my addr 172.21.15.28:0/3007731005 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9046800640 1 --2- 172.21.15.28:0/3007731005 >> v2:172.21.15.28:3300/0 conn(0x7f9048072190 0x7f90480cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9046800640 1 -- 172.21.15.28:0/3007731005 >> v1:172.21.15.151:6789/0 conn(0x7f90480b4800 legacy=0x7f90480ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045400640 1 -- 172.21.15.28:0/3007731005 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881556360 0 0) 0x7f904815bf90 con 0x7f90480b4420 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9046800640 1 -- 172.21.15.28:0/3007731005 >> v1:172.21.15.28:6789/0 conn(0x7f90480b4420 legacy=0x7f9048071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9046800640 1 -- 172.21.15.28:0/3007731005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904815b790 con 0x7f9048072190 2024-03-21T08:42:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9046800640 1 --2- 172.21.15.28:0/3007731005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9048072190 0x7f90480cd300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9040009d90 tx=0x7f904001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca44ff5e46a24b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045400640 1 -- 172.21.15.28:0/3007731005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90400b2070 con 0x7f9048072190 2024-03-21T08:42:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045400640 1 -- 172.21.15.28:0/3007731005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90400a59a0 con 0x7f9048072190 2024-03-21T08:42:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.435+0000 7f9045400640 1 -- 172.21.15.28:0/3007731005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90400a5d70 con 0x7f9048072190 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- 172.21.15.28:0/3007731005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9048072190 msgr2=0x7f90480cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 --2- 172.21.15.28:0/3007731005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9048072190 0x7f90480cd300 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9040009d90 tx=0x7f904001c940 comp rx=0 tx=0).stop 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- 172.21.15.28:0/3007731005 shutdown_connections 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 --2- 172.21.15.28:0/3007731005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9048072190 0x7f90480cd300 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- 172.21.15.28:0/3007731005 >> 172.21.15.28:0/3007731005 conn(0x7f90480674f0 msgr2=0x7f904806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- 172.21.15.28:0/3007731005 shutdown_connections 2024-03-21T08:42:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- 172.21.15.28:0/3007731005 wait complete. 2024-03-21T08:42:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 Processor -- start 2024-03-21T08:42:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- start start 2024-03-21T08:42:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9048072190 0x7f9048143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9048145700 0x7f9048147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9048168140 con 0x7f90480b4420 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9048167fc0 con 0x7f9048072190 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f904ca00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90481682c0 con 0x7f9048145700 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f903fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f903fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39528/0 (socket says 172.21.15.28:39528) 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f903fe00640 1 -- 172.21.15.28:0/1664162955 learned_addr learned my addr 172.21.15.28:0/1664162955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f9046800640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9048145700 0x7f9048147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.439+0000 7f9045e00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9048072190 0x7f9048143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 -- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9048145700 msgr2=0x7f9048147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9048145700 0x7f9048147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 -- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9048072190 msgr2=0x7f9048143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9048072190 0x7f9048143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 -- 172.21.15.28:0/1664162955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904814b580 con 0x7f90480b4420 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903fe00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f903000c3c0 tx=0x7f903000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030017070 con 0x7f90480b4420 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90300052d0 con 0x7f90480b4420 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030003820 con 0x7f90480b4420 2024-03-21T08:42:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9048145480 con 0x7f90480b4420 2024-03-21T08:42:25.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9048144ec0 con 0x7f90480b4420 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9048073430 con 0x7f90480b4420 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f903001a070 con 0x7f90480b4420 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 0x7f9024069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.443+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9030085090 con 0x7f90480b4420 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.447+0000 7f9045e00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 0x7f9024069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.447+0000 7f9045e00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 0x7f9024069fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9038000f70 tx=0x7f9038007bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.447+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9030005480 con 0x7f90480b4420 2024-03-21T08:42:25.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.675+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f904814b750 con 0x7f90480b4420 2024-03-21T08:42:25.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.675+0000 7f903e000640 1 -- 172.21.15.28:0/1664162955 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9030055c50 con 0x7f90480b4420 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 msgr2=0x7f9024069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 0x7f9024069fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9038000f70 tx=0x7f9038007bc0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 msgr2=0x7f904814dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f903000c3c0 tx=0x7f903000c880 comp rx=0 tx=0).stop 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 shutdown_connections 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9024067b20 0x7f9024069fe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9048145700 0x7f9048147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90480b4420 0x7f904814dc40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 --2- 172.21.15.28:0/1664162955 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9048072190 0x7f9048143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 >> 172.21.15.28:0/1664162955 conn(0x7f90480674f0 msgr2=0x7f9048154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 shutdown_connections 2024-03-21T08:42:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.679+0000 7f904ca00640 1 -- 172.21.15.28:0/1664162955 wait complete. 2024-03-21T08:42:25.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-21T08:42:25.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.839+0000 7fb14e200640 1 Processor -- start 2024-03-21T08:42:25.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.839+0000 7fb14e200640 1 -- start start 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1480b4420 0x7fb148071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 0x7fb148071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 0x7fb1480cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14815bf90 con 0x7fb1480b4420 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14815b1d0 con 0x7fb1480ccbf0 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb14e200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815b4b0 con 0x7fb1480b4800 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1480b4420 0x7fb148071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1480b4420 0x7fb148071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39536/0 (socket says 172.21.15.28:39536) 2024-03-21T08:42:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 -- 172.21.15.28:0/4050072037 learned_addr learned my addr 172.21.15.28:0/4050072037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147e00640 1 --2- 172.21.15.28:0/4050072037 >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 0x7fb1480cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb13ea00640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 0x7fb148071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 -- 172.21.15.28:0/4050072037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 msgr2=0x7fb148071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 0x7fb148071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 -- 172.21.15.28:0/4050072037 >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 msgr2=0x7fb1480cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 --2- 172.21.15.28:0/4050072037 >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 0x7fb1480cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147400640 1 -- 172.21.15.28:0/4050072037 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14815b790 con 0x7fb1480b4420 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb13ea00640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 0x7fb148071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.843+0000 7fb147e00640 1 --2- 172.21.15.28:0/4050072037 >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 0x7fb1480cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb147400640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148071a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb148152120 tx=0x7fb13801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1aa11b5c23cb893 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb146a00640 1 -- 172.21.15.28:0/4050072037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1380aa430 con 0x7fb1480b4420 2024-03-21T08:42:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb146a00640 1 -- 172.21.15.28:0/4050072037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1380af070 con 0x7fb1480b4420 2024-03-21T08:42:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb146a00640 1 -- 172.21.15.28:0/4050072037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1380aa7f0 con 0x7fb1480b4420 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- 172.21.15.28:0/4050072037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 msgr2=0x7fb148071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148071a80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb148152120 tx=0x7fb13801c4f0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- 172.21.15.28:0/4050072037 shutdown_connections 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 --2- 172.21.15.28:0/4050072037 >> v2:172.21.15.151:3300/0 conn(0x7fb1480ccbf0 0x7fb1480cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb1480b4800 0x7fb148071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 --2- 172.21.15.28:0/4050072037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148071a80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- 172.21.15.28:0/4050072037 >> 172.21.15.28:0/4050072037 conn(0x7fb1480674f0 msgr2=0x7fb14806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- 172.21.15.28:0/4050072037 shutdown_connections 2024-03-21T08:42:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- 172.21.15.28:0/4050072037 wait complete. 2024-03-21T08:42:25.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 Processor -- start 2024-03-21T08:42:25.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.847+0000 7fb14e200640 1 -- start start 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 0x7fb1480c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb14814b4e0 0x7fb14814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815efb0 con 0x7fb1480b4420 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815ee30 con 0x7fb1480ccbf0 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815f130 con 0x7fb14814b4e0 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39552/0 (socket says 172.21.15.28:39552) 2024-03-21T08:42:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb147400640 1 -- 172.21.15.28:0/1314962646 learned_addr learned my addr 172.21.15.28:0/1314962646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 0x7fb1480c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb147e00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb14814b4e0 0x7fb14814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 -- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb14814b4e0 msgr2=0x7fb14814ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb14814b4e0 0x7fb14814ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 -- 172.21.15.28:0/1314962646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 msgr2=0x7fb148160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 -- 172.21.15.28:0/1314962646 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1481610a0 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb147400640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13ea00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 0x7fb1480c9930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb13000e560 tx=0x7fb13000ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130017070 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb130013630 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13000f040 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148161380 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1480c9130 con 0x7fb1480ccbf0 2024-03-21T08:42:25.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.851+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb148073430 con 0x7fb1480ccbf0 2024-03-21T08:42:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.855+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb13001a070 con 0x7fb1480ccbf0 2024-03-21T08:42:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.855+0000 7fb13fe00640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 0x7fb12406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.855+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb130085740 con 0x7fb1480ccbf0 2024-03-21T08:42:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.855+0000 7fb147400640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 0x7fb12406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.855+0000 7fb147400640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 0x7fb12406a010 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb13801c530 tx=0x7fb1380a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:25.859+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb130055cb0 con 0x7fb1480ccbf0 2024-03-21T08:42:26.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.079+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fb148160710 con 0x7fb1480ccbf0 2024-03-21T08:42:26.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.087+0000 7fb13fe00640 1 -- 172.21.15.28:0/1314962646 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb130085070 con 0x7fb1480ccbf0 2024-03-21T08:42:26.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 msgr2=0x7fb12406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 0x7fb12406a010 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb13801c530 tx=0x7fb1380a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:26.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 msgr2=0x7fb1480c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 0x7fb1480c9930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb13000e560 tx=0x7fb13000ea20 comp rx=0 tx=0).stop 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 shutdown_connections 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb124067b50 0x7fb12406a010 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb14814b4e0 0x7fb14814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb1480ccbf0 0x7fb1480c9930 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 --2- 172.21.15.28:0/1314962646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1480b4420 0x7fb148160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 >> 172.21.15.28:0/1314962646 conn(0x7fb1480674f0 msgr2=0x7fb148071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 shutdown_connections 2024-03-21T08:42:26.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.091+0000 7fb14e200640 1 -- 172.21.15.28:0/1314962646 wait complete. 2024-03-21T08:42:26.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-21T08:42:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.251+0000 7ff30b800640 1 Processor -- start 2024-03-21T08:42:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.251+0000 7ff30b800640 1 -- start start 2024-03-21T08:42:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff30b800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff30406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff30b800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff3040cde40 0x7ff304071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff30b800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3040723a0 con 0x7ff30415cbb0 2024-03-21T08:42:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff30b800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30415c720 con 0x7ff3040cde40 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff30b800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040ce5b0 con 0x7ff304071a80 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff309800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff30415cbb0 0x7ff3040cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49800/0 (socket says 172.21.15.28:49800) 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff309800640 1 -- 172.21.15.28:0/244873507 learned_addr learned my addr 172.21.15.28:0/244873507 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> v2:172.21.15.151:3300/0 conn(0x7ff3040cde40 0x7ff304071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff308e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff30406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142232948 0 0) 0x7ff3040723a0 con 0x7ff30415cbb0 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec003530 con 0x7ff30415cbb0 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3977907353 0 0) 0x7ff2ec003530 con 0x7ff30415cbb0 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 msgr2=0x7ff30406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff30406f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 >> v2:172.21.15.151:3300/0 conn(0x7ff3040cde40 msgr2=0x7ff304071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 --2- 172.21.15.28:0/244873507 >> v2:172.21.15.151:3300/0 conn(0x7ff3040cde40 0x7ff304071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040ce890 con 0x7ff30415cbb0 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7ff2fc001a00 con 0x7ff30415cbb0 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 >> v1:172.21.15.28:6789/0 conn(0x7ff30415cbb0 legacy=0x7ff3040cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2ec0039c0 0x7ff2ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ec004810 0x7ff2ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff2ec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30415c720 con 0x7ff2ec004810 2024-03-21T08:42:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040ce5b0 con 0x7ff2ec0039c0 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec00b9d0 con 0x7ff304071a80 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> v2:172.21.15.151:3300/0 conn(0x7ff3040cde40 0x7ff304071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ec004810 0x7ff2ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff309800640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff2ec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff308e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2ec0039c0 0x7ff2ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 -- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 msgr2=0x7ff2ec0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff304071a80 0x7ff2ec0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 -- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2ec0039c0 msgr2=0x7ff2ec003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff2ec0039c0 0x7ff2ec003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.255+0000 7ff303e00640 1 -- 172.21.15.28:0/244873507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec003530 con 0x7ff2ec004810 2024-03-21T08:42:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff303e00640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ec004810 0x7ff2ec004c90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009cc0 tx=0x7ff2f001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a551fbf0bdc43a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0009860 con 0x7ff2ec004810 2024-03-21T08:42:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f0004020 con 0x7ff2ec004810 2024-03-21T08:42:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff303400640 1 -- 172.21.15.28:0/244873507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0004410 con 0x7ff2ec004810 2024-03-21T08:42:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- 172.21.15.28:0/244873507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ec004810 msgr2=0x7ff2ec004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 --2- 172.21.15.28:0/244873507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2ec004810 0x7ff2ec004c90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009cc0 tx=0x7ff2f001cc20 comp rx=0 tx=0).stop 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff309800640 1 -- 172.21.15.28:0/244873507 reap_dead start 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- 172.21.15.28:0/244873507 shutdown_connections 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- 172.21.15.28:0/244873507 >> 172.21.15.28:0/244873507 conn(0x7ff3040674f0 msgr2=0x7ff30406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- 172.21.15.28:0/244873507 shutdown_connections 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- 172.21.15.28:0/244873507 wait complete. 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 Processor -- start 2024-03-21T08:42:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.259+0000 7ff30b800640 1 -- start start 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff30415b6d0 0x7ff30413d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff30413e380 0x7ff304140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30406f0a0 con 0x7ff3041411f0 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040ce220 con 0x7ff30413e380 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30406f220 con 0x7ff30415b6d0 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39568/0 (socket says 172.21.15.28:39568) 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 -- 172.21.15.28:0/3002520164 learned_addr learned my addr 172.21.15.28:0/3002520164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff303e00640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff30413e380 0x7ff304140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff308e00640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff30415b6d0 0x7ff30413d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 -- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff30415b6d0 msgr2=0x7ff30413d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff30415b6d0 0x7ff30413d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 -- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff30413e380 msgr2=0x7ff304140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff30413e380 0x7ff304140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 -- 172.21.15.28:0/3002520164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040b1590 con 0x7ff3041411f0 2024-03-21T08:42:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff309800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff3040b50d0 tx=0x7ff2fc007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc00aac0 con 0x7ff3041411f0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc018070 con 0x7ff3041411f0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc0133f0 con 0x7ff3041411f0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3041f5dc0 con 0x7ff3041411f0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff304140790 con 0x7ff3041411f0 2024-03-21T08:42:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff304140990 con 0x7ff3041411f0 2024-03-21T08:42:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.263+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff2fc013590 con 0x7ff3041411f0 2024-03-21T08:42:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.267+0000 7ff300c00640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 0x7ff2e406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.267+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff2fc086070 con 0x7ff3041411f0 2024-03-21T08:42:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.267+0000 7ff308e00640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 0x7ff2e406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.267+0000 7ff308e00640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 0x7ff2e406a010 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff304164870 tx=0x7ff2f8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.271+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff304140990 con 0x7ff3041411f0 2024-03-21T08:42:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.491+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff304073430 con 0x7ff3041411f0 2024-03-21T08:42:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff300c00640 1 -- 172.21.15.28:0/3002520164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff2fc056c40 con 0x7ff3041411f0 2024-03-21T08:42:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 msgr2=0x7ff2e406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 0x7ff2e406a010 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff304164870 tx=0x7ff2f8009190 comp rx=0 tx=0).stop 2024-03-21T08:42:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 msgr2=0x7ff304162970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff3040b50d0 tx=0x7ff2fc007970 comp rx=0 tx=0).stop 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 shutdown_connections 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3041411f0 0x7ff304162970 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff30413e380 0x7ff304140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff2e4067b50 0x7ff2e406a010 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 --2- 172.21.15.28:0/3002520164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff30415b6d0 0x7ff30413d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 >> 172.21.15.28:0/3002520164 conn(0x7ff3040674f0 msgr2=0x7ff304142b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 shutdown_connections 2024-03-21T08:42:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.495+0000 7ff30b800640 1 -- 172.21.15.28:0/3002520164 wait complete. 2024-03-21T08:42:26.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:26.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 Processor -- start 2024-03-21T08:42:26.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 -- start start 2024-03-21T08:42:26.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c099b80 0x7f4a9c099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c0cdb50 con 0x7f4a9c15f260 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c09a520 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa3a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0c5530 con 0x7f4a9c099b80 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39580/0 (socket says 172.21.15.28:39580) 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 -- 172.21.15.28:0/624838576 learned_addr learned my addr 172.21.15.28:0/624838576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4a93e00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c099b80 0x7f4a9c099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 -- 172.21.15.28:0/624838576 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c099b80 msgr2=0x7f4a9c099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c099b80 0x7f4a9c099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 -- 172.21.15.28:0/624838576 >> v1:172.21.15.151:6789/0 conn(0x7f4a9c15f260 legacy=0x7f4a9c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.659+0000 7f4aa1a00640 1 -- 172.21.15.28:0/624838576 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0cde30 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa1a00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009e50 tx=0x7f4a9401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f32f64498c2edca server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4a9be00640 1 -- 172.21.15.28:0/624838576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940a5830 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4a9be00640 1 -- 172.21.15.28:0/624838576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a940a59d0 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4a9be00640 1 -- 172.21.15.28:0/624838576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940a5da0 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 -- 172.21.15.28:0/624838576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c0c5f80 msgr2=0x7f4a9c0c63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009e50 tx=0x7f4a9401c970 comp rx=0 tx=0).stop 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 -- 172.21.15.28:0/624838576 shutdown_connections 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c0c5f80 0x7f4a9c0c63e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/624838576 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c099b80 0x7f4a9c099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 -- 172.21.15.28:0/624838576 >> 172.21.15.28:0/624838576 conn(0x7f4a9c0b34d0 msgr2=0x7f4a9c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 -- 172.21.15.28:0/624838576 shutdown_connections 2024-03-21T08:42:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.663+0000 7f4aa3a00640 1 -- 172.21.15.28:0/624838576 wait complete. 2024-03-21T08:42:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 Processor -- start 2024-03-21T08:42:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- start start 2024-03-21T08:42:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c0c5f80 0x7f4a9c156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 0x7f4a9c1648b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0cd7a0 con 0x7f4a9c099b80 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c15f640 con 0x7f4a9c15f260 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0ca760 con 0x7f4a9c0c5f80 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39596/0 (socket says 172.21.15.28:39596) 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 -- 172.21.15.28:0/2157649457 learned_addr learned my addr 172.21.15.28:0/2157649457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 0x7f4a9c1648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4a93e00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c0c5f80 0x7f4a9c156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 -- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c0c5f80 msgr2=0x7f4a9c156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c0c5f80 0x7f4a9c156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 -- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 msgr2=0x7f4a9c1648b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 0x7f4a9c1648b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 -- 172.21.15.28:0/2157649457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0cb130 con 0x7f4a9c099b80 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 0x7f4a9c1648b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa1000640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c370 tx=0x7f4a8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0052d0 con 0x7f4a9c099b80 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c005470 con 0x7f4a9c099b80 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c007260 con 0x7f4a9c099b80 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c0c9e50 con 0x7f4a9c099b80 2024-03-21T08:42:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c0ca3f0 con 0x7f4a9c099b80 2024-03-21T08:42:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.667+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c04f410 con 0x7f4a9c099b80 2024-03-21T08:42:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.671+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4a8c007400 con 0x7f4a9c099b80 2024-03-21T08:42:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.671+0000 7f4a99600640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 0x7f4a78069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.671+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4a8c0851c0 con 0x7f4a9c099b80 2024-03-21T08:42:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.671+0000 7f4a93e00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 0x7f4a78069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.671+0000 7f4a93e00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 0x7f4a78069fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c1fa400 tx=0x7f4a84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.675+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4a9c0cb130 con 0x7f4a9c099b80 2024-03-21T08:42:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.847+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a9c0cfc60 con 0x7f4a9c099b80 2024-03-21T08:42:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.847+0000 7f4a99600640 1 -- 172.21.15.28:0/2157649457 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13176 (secure 0 0 0) 0x7f4a8c055d80 con 0x7f4a9c099b80 2024-03-21T08:42:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:26.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:23.654195+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 msgr2=0x7f4a78069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 0x7f4a78069fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c1fa400 tx=0x7f4a84008040 comp rx=0 tx=0).stop 2024-03-21T08:42:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 msgr2=0x7f4a9c0cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c370 tx=0x7f4a8c00c830 comp rx=0 tx=0).stop 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 shutdown_connections 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a9c15f260 0x7f4a9c1648b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4a78067b20 0x7f4a78069fe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a9c0c5f80 0x7f4a9c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 --2- 172.21.15.28:0/2157649457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c099b80 0x7f4a9c0cabf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 >> 172.21.15.28:0/2157649457 conn(0x7f4a9c0b34d0 msgr2=0x7f4a9c0cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 shutdown_connections 2024-03-21T08:42:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:26.851+0000 7f4aa3a00640 1 -- 172.21.15.28:0/2157649457 wait complete. 2024-03-21T08:42:26.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:42:26.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 Processor -- start 2024-03-21T08:42:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 -- start start 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79f415b970 0x7f79f4070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 0x7f79f406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40ce370 con 0x7f79f415b970 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40737c0 con 0x7f79f406b170 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4072fb0 con 0x7f79f415f4b0 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79f415b970 0x7f79f4070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79f415b970 0x7f79f4070b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39606/0 (socket says 172.21.15.28:39606) 2024-03-21T08:42:27.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 -- 172.21.15.28:0/206082601 learned_addr learned my addr 172.21.15.28:0/206082601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79ebe00640 1 --2- 172.21.15.28:0/206082601 >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 0x7f79f406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3400640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 -- 172.21.15.28:0/206082601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 msgr2=0x7f79f415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 -- 172.21.15.28:0/206082601 >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 msgr2=0x7f79f406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 --2- 172.21.15.28:0/206082601 >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 0x7f79f406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f2a00640 1 -- 172.21.15.28:0/206082601 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f415bd50 con 0x7f79f415b970 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79ebe00640 1 --2- 172.21.15.28:0/206082601 >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 0x7f79f406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.015+0000 7f79f3400640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:27.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f2a00640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f415b970 0x7f79f4070b90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f79f4072870 tx=0x7f79e401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c6e2780f649f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f2000640 1 -- 172.21.15.28:0/206082601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40b0070 con 0x7f79f415b970 2024-03-21T08:42:27.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f2000640 1 -- 172.21.15.28:0/206082601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e40ab600 con 0x7f79f415b970 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f2000640 1 -- 172.21.15.28:0/206082601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40ab9d0 con 0x7f79f415b970 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 -- 172.21.15.28:0/206082601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f415b970 msgr2=0x7f79f4070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f415b970 0x7f79f4070b90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f79f4072870 tx=0x7f79e401c520 comp rx=0 tx=0).stop 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 -- 172.21.15.28:0/206082601 shutdown_connections 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f415f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 --2- 172.21.15.28:0/206082601 >> v2:172.21.15.151:3300/0 conn(0x7f79f406b170 0x7f79f406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 --2- 172.21.15.28:0/206082601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f415b970 0x7f79f4070b90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 -- 172.21.15.28:0/206082601 >> 172.21.15.28:0/206082601 conn(0x7f79f40b23d0 msgr2=0x7f79f40b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 -- 172.21.15.28:0/206082601 shutdown_connections 2024-03-21T08:42:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.019+0000 7f79f3e00640 1 -- 172.21.15.28:0/206082601 wait complete. 2024-03-21T08:42:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 Processor -- start 2024-03-21T08:42:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- start start 2024-03-21T08:42:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 0x7f79f41512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f4151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4070840 con 0x7f79f406b170 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40cdbb0 con 0x7f79f415b970 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40710d0 con 0x7f79f415f4b0 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39618/0 (socket says 172.21.15.28:39618) 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 -- 172.21.15.28:0/1275097506 learned_addr learned my addr 172.21.15.28:0/1275097506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3400640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f4151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79ebe00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 0x7f79f41512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 -- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 msgr2=0x7f79f4151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f4151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 -- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 msgr2=0x7f79f41512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 0x7f79f41512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 -- 172.21.15.28:0/1275097506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f415cd40 con 0x7f79f406b170 2024-03-21T08:42:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79ebe00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 0x7f79f41512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f2a00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f79e40ae040 tx=0x7f79e40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40b0040 con 0x7f79f406b170 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e40b9070 con 0x7f79f406b170 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f415d020 con 0x7f79f406b170 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e40b4680 con 0x7f79f406b170 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f415d5e0 con 0x7f79f406b170 2024-03-21T08:42:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.023+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f404f410 con 0x7f79f406b170 2024-03-21T08:42:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.027+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f79e40b4820 con 0x7f79f406b170 2024-03-21T08:42:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.027+0000 7f79eaa00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 0x7f79d0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.027+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f79e411faa0 con 0x7f79f406b170 2024-03-21T08:42:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.027+0000 7f79ebe00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 0x7f79d0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.027+0000 7f79ebe00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 0x7f79d0069fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f79f41526e0 tx=0x7f79dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.031+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f79f415bd50 con 0x7f79f406b170 2024-03-21T08:42:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.199+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79f41662d0 con 0x7f79f406b170 2024-03-21T08:42:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.199+0000 7f79eaa00640 1 -- 172.21.15.28:0/1275097506 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79e4016110 con 0x7f79f406b170 2024-03-21T08:42:27.202 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 msgr2=0x7f79d0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 0x7f79d0069fe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f79f41526e0 tx=0x7f79dc008040 comp rx=0 tx=0).stop 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 msgr2=0x7f79f415dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f79e40ae040 tx=0x7f79e40192c0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 shutdown_connections 2024-03-21T08:42:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f79d0067b20 0x7f79d0069fe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f79f415f4b0 0x7f79f4151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f79f415b970 0x7f79f41512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 --2- 172.21.15.28:0/1275097506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79f406b170 0x7f79f415dde0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 >> 172.21.15.28:0/1275097506 conn(0x7f79f40b23d0 msgr2=0x7f79f41573d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 shutdown_connections 2024-03-21T08:42:27.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.203+0000 7f79f3e00640 1 -- 172.21.15.28:0/1275097506 wait complete. 2024-03-21T08:42:27.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-21T08:42:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.359+0000 7f963ee00640 1 Processor -- start 2024-03-21T08:42:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.359+0000 7f963ee00640 1 -- start start 2024-03-21T08:42:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ee00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 0x7f96380cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f96380b5520 0x7f963815c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963815bf00 con 0x7f96380b34f0 2024-03-21T08:42:27.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380ce6e0 con 0x7f96380b5520 2024-03-21T08:42:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ee00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380b3980 con 0x7f96380af060 2024-03-21T08:42:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f96380b5520 0x7f963815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f962fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f96380b34f0 0x7f96380b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49802/0 (socket says 172.21.15.28:49802) 2024-03-21T08:42:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f962fe00640 1 -- 172.21.15.28:0/2579914081 learned_addr learned my addr 172.21.15.28:0/2579914081 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f9637e00640 1 --2- 172.21.15.28:0/2579914081 >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 0x7f96380cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ce00640 1 -- 172.21.15.28:0/2579914081 >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 msgr2=0x7f96380cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ce00640 1 --2- 172.21.15.28:0/2579914081 >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 0x7f96380cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ce00640 1 -- 172.21.15.28:0/2579914081 >> v1:172.21.15.28:6789/0 conn(0x7f96380b34f0 legacy=0x7f96380b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f963ce00640 1 -- 172.21.15.28:0/2579914081 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380cdad0 con 0x7f96380b5520 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f9637400640 1 -- 172.21.15.28:0/2579914081 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758827810 0 0) 0x7f963815bf00 con 0x7f96380b34f0 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.363+0000 7f9637e00640 1 --2- 172.21.15.28:0/2579914081 >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 0x7f96380cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ce00640 1 --2- 172.21.15.28:0/2579914081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96380b5520 0x7f963815c200 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9638073140 tx=0x7f963001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38d2dfb8559b227 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f9637400640 1 -- 172.21.15.28:0/2579914081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300b2070 con 0x7f96380b5520 2024-03-21T08:42:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f9637400640 1 -- 172.21.15.28:0/2579914081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300a5850 con 0x7f96380b5520 2024-03-21T08:42:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f9637400640 1 -- 172.21.15.28:0/2579914081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300a5c20 con 0x7f96380b5520 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- 172.21.15.28:0/2579914081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96380b5520 msgr2=0x7f963815c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- 172.21.15.28:0/2579914081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96380b5520 0x7f963815c200 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9638073140 tx=0x7f963001c8a0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- 172.21.15.28:0/2579914081 shutdown_connections 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- 172.21.15.28:0/2579914081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96380b5520 0x7f963815c200 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- 172.21.15.28:0/2579914081 >> v2:172.21.15.151:3300/0 conn(0x7f96380af060 0x7f96380cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- 172.21.15.28:0/2579914081 >> 172.21.15.28:0/2579914081 conn(0x7f96380cc230 msgr2=0x7f96380cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- 172.21.15.28:0/2579914081 shutdown_connections 2024-03-21T08:42:27.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- 172.21.15.28:0/2579914081 wait complete. 2024-03-21T08:42:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 Processor -- start 2024-03-21T08:42:27.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- start start 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96380b34f0 0x7f9638135b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 0x7f963815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963815d0d0 con 0x7f963815aaf0 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963815cf50 con 0x7f96380b34f0 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.367+0000 7f963ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963815d250 con 0x7f96380b5520 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39628/0 (socket says 172.21.15.28:39628) 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 -- 172.21.15.28:0/1880929642 learned_addr learned my addr 172.21.15.28:0/1880929642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f9637e00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96380b34f0 0x7f9638135b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f962fe00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 0x7f963815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 -- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 msgr2=0x7f963815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 0x7f963815a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 -- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96380b34f0 msgr2=0x7f9638135b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96380b34f0 0x7f9638135b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 -- 172.21.15.28:0/1880929642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638155df0 con 0x7f963815aaf0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f962fe00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 0x7f963815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ce00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f963001a4d0 tx=0x7f9630019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300b2050 con 0x7f963815aaf0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300be070 con 0x7f963815aaf0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300ada90 con 0x7f963815aaf0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96381560d0 con 0x7f963815aaf0 2024-03-21T08:42:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638156690 con 0x7f963815aaf0 2024-03-21T08:42:27.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96381569c0 con 0x7f963815aaf0 2024-03-21T08:42:27.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.371+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f96300a56d0 con 0x7f963815aaf0 2024-03-21T08:42:27.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.375+0000 7f9634c00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 0x7f961406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.375+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9630120e70 con 0x7f963815aaf0 2024-03-21T08:42:27.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.375+0000 7f9637e00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 0x7f961406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.375+0000 7f9637e00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 0x7f961406a010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9628009950 tx=0x7f9628008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.379+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9630016520 con 0x7f963815aaf0 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.539+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f963804f410 con 0x7f963815aaf0 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.539+0000 7f9634c00640 1 -- 172.21.15.28:0/1880929642 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f96300f1ad0 con 0x7f963815aaf0 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:42:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-03-21T08:42:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-03-21T08:42:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:27.544 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 msgr2=0x7f961406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 0x7f961406a010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9628009950 tx=0x7f9628008040 comp rx=0 tx=0).stop 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 msgr2=0x7f963814cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f963001a4d0 tx=0x7f9630019640 comp rx=0 tx=0).stop 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 shutdown_connections 2024-03-21T08:42:27.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963815aaf0 0x7f963814cd10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9614067b50 0x7f961406a010 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96380b5520 0x7f963815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 --2- 172.21.15.28:0/1880929642 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96380b34f0 0x7f9638135b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 >> 172.21.15.28:0/1880929642 conn(0x7f96380cc230 msgr2=0x7f96380b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 shutdown_connections 2024-03-21T08:42:27.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.543+0000 7f963ee00640 1 -- 172.21.15.28:0/1880929642 wait complete. 2024-03-21T08:42:27.564 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-21T08:42:27.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-21T08:42:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 Processor -- start 2024-03-21T08:42:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 -- start start 2024-03-21T08:42:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 0x7f178415c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178406da60 con 0x7f178415c150 2024-03-21T08:42:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178415cc80 con 0x7f1784072130 2024-03-21T08:42:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f178a400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178415cf60 con 0x7f17840ce6d0 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f1783400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f1783e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 0x7f178415c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f177be00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f178415c150 0x7f1784071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:57938/0 (socket says 172.21.15.28:57938) 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f1783400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54968/0 (socket says 172.21.15.28:54968) 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.707+0000 7f177be00640 1 -- 172.21.15.28:0/227155712 learned_addr learned my addr 172.21.15.28:0/227155712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783400640 1 -- 172.21.15.28:0/227155712 >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 msgr2=0x7f178415c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783400640 1 --2- 172.21.15.28:0/227155712 >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 0x7f178415c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783400640 1 -- 172.21.15.28:0/227155712 >> v1:172.21.15.151:6789/0 conn(0x7f178415c150 legacy=0x7f1784071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783400640 1 -- 172.21.15.28:0/227155712 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178406dd20 con 0x7f17840ce6d0 2024-03-21T08:42:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783e00640 1 --2- 172.21.15.28:0/227155712 >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 0x7f178415c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1783400640 1 --2- 172.21.15.28:0/227155712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17740097c0 tx=0x7f177401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=495d08b8d05c40ae server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1782a00640 1 -- 172.21.15.28:0/227155712 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17740b0070 con 0x7f17840ce6d0 2024-03-21T08:42:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1782a00640 1 -- 172.21.15.28:0/227155712 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17740ab600 con 0x7f17840ce6d0 2024-03-21T08:42:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f1782a00640 1 -- 172.21.15.28:0/227155712 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17740ab9d0 con 0x7f17840ce6d0 2024-03-21T08:42:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 -- 172.21.15.28:0/227155712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 msgr2=0x7f178406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 --2- 172.21.15.28:0/227155712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17740097c0 tx=0x7f177401c410 comp rx=0 tx=0).stop 2024-03-21T08:42:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 -- 172.21.15.28:0/227155712 shutdown_connections 2024-03-21T08:42:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 --2- 172.21.15.28:0/227155712 >> v2:172.21.15.151:3300/0 conn(0x7f1784072130 0x7f178415c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 --2- 172.21.15.28:0/227155712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178406e520 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.711+0000 7f178a400640 1 -- 172.21.15.28:0/227155712 >> 172.21.15.28:0/227155712 conn(0x7f17840674f0 msgr2=0x7f178406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- 172.21.15.28:0/227155712 shutdown_connections 2024-03-21T08:42:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- 172.21.15.28:0/227155712 wait complete. 2024-03-21T08:42:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 Processor -- start 2024-03-21T08:42:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- start start 2024-03-21T08:42:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 0x7f178414a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178414a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840ccd90 con 0x7f178415c150 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840ccc10 con 0x7f1784072130 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17840ccf10 con 0x7f17840ce6d0 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39630/0 (socket says 172.21.15.28:39630) 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 -- 172.21.15.28:0/1764205454 learned_addr learned my addr 172.21.15.28:0/1764205454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f177be00640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178414a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 0x7f178414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 -- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 msgr2=0x7f178414a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178414a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 -- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 msgr2=0x7f178414a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 0x7f178414a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 -- 172.21.15.28:0/1764205454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784151de0 con 0x7f178415c150 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 0x7f178414a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f1783e00640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f177c00efb0 tx=0x7f177c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c018020 con 0x7f178415c150 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c014480 con 0x7f178415c150 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c014850 con 0x7f178415c150 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.715+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17840b4180 con 0x7f178415c150 2024-03-21T08:42:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.719+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1784150050 con 0x7f178415c150 2024-03-21T08:42:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.719+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1784068110 con 0x7f178415c150 2024-03-21T08:42:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.719+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f177c01b020 con 0x7f178415c150 2024-03-21T08:42:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.719+0000 7f177b400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 0x7f1760069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.719+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f177c086b60 con 0x7f178415c150 2024-03-21T08:42:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.723+0000 7f1783400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 0x7f1760069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.723+0000 7f1783400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 0x7f1760069fe0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17740ad000 tx=0x7f1774008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.727+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f177c022030 con 0x7f178415c150 2024-03-21T08:42:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:27.887+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f17840741c0 con 0x7f178415c150 2024-03-21T08:42:28.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.655+0000 7f177b400640 1 -- 172.21.15.28:0/1764205454 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f177c010040 con 0x7f178415c150 2024-03-21T08:42:28.659 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-03-21T08:42:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 msgr2=0x7f1760069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 0x7f1760069fe0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17740ad000 tx=0x7f1774008040 comp rx=0 tx=0).stop 2024-03-21T08:42:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 msgr2=0x7f1784140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f177c00efb0 tx=0x7f177c00c490 comp rx=0 tx=0).stop 2024-03-21T08:42:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 shutdown_connections 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178415c150 0x7f1784140470 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1760067b20 0x7f1760069fe0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f17840ce6d0 0x7f178414a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 --2- 172.21.15.28:0/1764205454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1784072130 0x7f178414a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 >> 172.21.15.28:0/1764205454 conn(0x7f17840674f0 msgr2=0x7f17840afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 shutdown_connections 2024-03-21T08:42:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.659+0000 7f178a400640 1 -- 172.21.15.28:0/1764205454 wait complete. 2024-03-21T08:42:28.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-21T08:42:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.835+0000 7f7a89a00640 1 Processor -- start 2024-03-21T08:42:28.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.835+0000 7f7a89a00640 1 -- start start 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a89a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a84071a80 0x7f7a8406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a89a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a84071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a89a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a840723a0 con 0x7f7a8415cbb0 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a89a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8415c720 con 0x7f7a84071a80 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a89a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840ce5b0 con 0x7f7a840cde40 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a84071a80 0x7f7a8406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a84071a80 0x7f7a8406f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39642/0 (socket says 172.21.15.28:39642) 2024-03-21T08:42:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 -- 172.21.15.28:0/2479263166 learned_addr learned my addr 172.21.15.28:0/2479263166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:28.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 -- 172.21.15.28:0/2479263166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 msgr2=0x7f7a84071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:28.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 --2- 172.21.15.28:0/2479263166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a84071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 -- 172.21.15.28:0/2479263166 >> v1:172.21.15.28:6789/0 conn(0x7f7a8415cbb0 legacy=0x7f7a840cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.839+0000 7f7a82a00640 1 -- 172.21.15.28:0/2479263166 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840ce890 con 0x7f7a84071a80 2024-03-21T08:42:28.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a82a00640 1 --2- 172.21.15.28:0/2479263166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a84071a80 0x7f7a8406f0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009870 tx=0x7f7a7401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b239bff111ff8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:28.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a82000640 1 -- 172.21.15.28:0/2479263166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740aa620 con 0x7f7a84071a80 2024-03-21T08:42:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a82000640 1 -- 172.21.15.28:0/2479263166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a740af070 con 0x7f7a84071a80 2024-03-21T08:42:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a82000640 1 -- 172.21.15.28:0/2479263166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a740aa9e0 con 0x7f7a84071a80 2024-03-21T08:42:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- 172.21.15.28:0/2479263166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a84071a80 msgr2=0x7f7a8406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 --2- 172.21.15.28:0/2479263166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a84071a80 0x7f7a8406f0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009870 tx=0x7f7a7401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- 172.21.15.28:0/2479263166 shutdown_connections 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 --2- 172.21.15.28:0/2479263166 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a84071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 --2- 172.21.15.28:0/2479263166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a84071a80 0x7f7a8406f0a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- 172.21.15.28:0/2479263166 >> 172.21.15.28:0/2479263166 conn(0x7f7a840674f0 msgr2=0x7f7a8406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- 172.21.15.28:0/2479263166 shutdown_connections 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- 172.21.15.28:0/2479263166 wait complete. 2024-03-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 Processor -- start 2024-03-21T08:42:28.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.843+0000 7f7a89a00640 1 -- start start 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 0x7f7a8415b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a840ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84168f20 con 0x7f7a8415cbb0 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84144b20 con 0x7f7a84071a80 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a841690a0 con 0x7f7a840cde40 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39658/0 (socket says 172.21.15.28:39658) 2024-03-21T08:42:28.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 -- 172.21.15.28:0/86005872 learned_addr learned my addr 172.21.15.28:0/86005872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:28.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a82a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 0x7f7a8415b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 -- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 msgr2=0x7f7a840ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a840ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 -- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 msgr2=0x7f7a8415b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 0x7f7a8415b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 -- 172.21.15.28:0/86005872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8413c8d0 con 0x7f7a8415cbb0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a82a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 0x7f7a8415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a83400640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c730 tx=0x7f7a7c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c003b80 con 0x7f7a8415cbb0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c003d20 con 0x7f7a8415cbb0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a841415d0 con 0x7f7a8415cbb0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c005310 con 0x7f7a8415cbb0 2024-03-21T08:42:28.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8413cb90 con 0x7f7a8415cbb0 2024-03-21T08:42:28.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8404f4f0 con 0x7f7a8415cbb0 2024-03-21T08:42:28.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.847+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7a7c0056c0 con 0x7f7a8415cbb0 2024-03-21T08:42:28.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.851+0000 7f7a7b400640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 0x7f7a60069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:28.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.851+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f7a7c085f10 con 0x7f7a8415cbb0 2024-03-21T08:42:28.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.851+0000 7f7a82a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 0x7f7a60069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:28.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.851+0000 7f7a82a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 0x7f7a60069fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a740ae000 tx=0x7f7a74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:28.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:28.855+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7a7c056960 con 0x7f7a8415cbb0 2024-03-21T08:42:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.015+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7a84073410 con 0x7f7a8415cbb0 2024-03-21T08:42:29.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-21T08:42:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.659+0000 7f7a7b400640 1 -- 172.21.15.28:0/86005872 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7a7c056260 con 0x7f7a8415cbb0 2024-03-21T08:42:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 msgr2=0x7f7a60069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 0x7f7a60069fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a740ae000 tx=0x7f7a74008040 comp rx=0 tx=0).stop 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 msgr2=0x7f7a8413a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c730 tx=0x7f7a7c00cbf0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 shutdown_connections 2024-03-21T08:42:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8415cbb0 0x7f7a8413a330 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7a60067b20 0x7f7a60069fe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7a840cde40 0x7f7a840ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 --2- 172.21.15.28:0/86005872 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7a84071a80 0x7f7a8415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 >> 172.21.15.28:0/86005872 conn(0x7f7a840674f0 msgr2=0x7f7a8413b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 shutdown_connections 2024-03-21T08:42:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.663+0000 7f7a89a00640 1 -- 172.21.15.28:0/86005872 wait complete. 2024-03-21T08:42:29.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-21T08:42:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.847+0000 7faefe000640 1 Processor -- start 2024-03-21T08:42:29.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.847+0000 7faefe000640 1 -- start start 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefe000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef815ba50 0x7faef80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefe000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faef80cc170 0x7faef8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefe000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef815be30 con 0x7faef80ce320 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefe000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80ccea0 con 0x7faef80cc170 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefe000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80cc8e0 con 0x7faef815ba50 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faeefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faef80cc170 0x7faef8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefd600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faef80ce320 0x7faef80b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49818/0 (socket says 172.21.15.28:49818) 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefd600640 1 -- 172.21.15.28:0/2498546557 learned_addr learned my addr 172.21.15.28:0/2498546557 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faefcc00640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef815ba50 0x7faef80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faeefe00640 1 -- 172.21.15.28:0/2498546557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef815ba50 msgr2=0x7faef80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faeefe00640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef815ba50 0x7faef80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faeefe00640 1 -- 172.21.15.28:0/2498546557 >> v1:172.21.15.28:6789/0 conn(0x7faef80ce320 legacy=0x7faef80b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faeefe00640 1 -- 172.21.15.28:0/2498546557 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80ccbc0 con 0x7faef80cc170 2024-03-21T08:42:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.851+0000 7faef7e00640 1 -- 172.21.15.28:0/2498546557 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214469328 0 0) 0x7faef815be30 con 0x7faef80ce320 2024-03-21T08:42:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faeefe00640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef80cc170 0x7faef8071bb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faee00097f0 tx=0x7faee001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ddf95556c2e14c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faef7e00640 1 -- 172.21.15.28:0/2498546557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00a7070 con 0x7faef80cc170 2024-03-21T08:42:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faef7e00640 1 -- 172.21.15.28:0/2498546557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee00b0070 con 0x7faef80cc170 2024-03-21T08:42:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faef7e00640 1 -- 172.21.15.28:0/2498546557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00ab890 con 0x7faef80cc170 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- 172.21.15.28:0/2498546557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef80cc170 msgr2=0x7faef8071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef80cc170 0x7faef8071bb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faee00097f0 tx=0x7faee001c470 comp rx=0 tx=0).stop 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- 172.21.15.28:0/2498546557 shutdown_connections 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef80cc170 0x7faef8071bb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- 172.21.15.28:0/2498546557 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef815ba50 0x7faef80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- 172.21.15.28:0/2498546557 >> 172.21.15.28:0/2498546557 conn(0x7faef806fa90 msgr2=0x7faef806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- 172.21.15.28:0/2498546557 shutdown_connections 2024-03-21T08:42:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- 172.21.15.28:0/2498546557 wait complete. 2024-03-21T08:42:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 Processor -- start 2024-03-21T08:42:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- start start 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faef80cc170 0x7faef814c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef80ce320 0x7faef8165330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80ceb80 con 0x7faef815ba50 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80cc5b0 con 0x7faef80cc170 2024-03-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.855+0000 7faefe000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80720f0 con 0x7faef80ce320 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39672/0 (socket says 172.21.15.28:39672) 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 -- 172.21.15.28:0/3576038598 learned_addr learned my addr 172.21.15.28:0/3576038598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefcc00640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faef80cc170 0x7faef814c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faeefe00640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef80ce320 0x7faef8165330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 -- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef80ce320 msgr2=0x7faef8165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef80ce320 0x7faef8165330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 -- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faef80cc170 msgr2=0x7faef814c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faef80cc170 0x7faef814c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 -- 172.21.15.28:0/3576038598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8164570 con 0x7faef815ba50 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefd600640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faef000c7a0 tx=0x7faef000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0018070 con 0x7faef815ba50 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef000a640 con 0x7faef815ba50 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000aab0 con 0x7faef815ba50 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef8164850 con 0x7faef815ba50 2024-03-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef8164e10 con 0x7faef815ba50 2024-03-21T08:42:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef804f410 con 0x7faef815ba50 2024-03-21T08:42:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.859+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faef001b070 con 0x7faef815ba50 2024-03-21T08:42:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.863+0000 7faef5600640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 0x7faed4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.863+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7faef0085a00 con 0x7faef815ba50 2024-03-21T08:42:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.863+0000 7faefcc00640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 0x7faed4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.863+0000 7faefcc00640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 0x7faed4069fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faee800b930 tx=0x7faee8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:29.863+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faef00562c0 con 0x7faef815ba50 2024-03-21T08:42:30.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.063+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faef80cbcc0 con 0x7faef815ba50 2024-03-21T08:42:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.675+0000 7faef5600640 1 -- 172.21.15.28:0/3576038598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7faef0055bc0 con 0x7faef815ba50 2024-03-21T08:42:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 5 and data pool 6 2024-03-21T08:42:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.687+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 msgr2=0x7faed4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.687+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 0x7faed4069fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faee800b930 tx=0x7faee8008040 comp rx=0 tx=0).stop 2024-03-21T08:42:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.687+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 msgr2=0x7faef81537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.687+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faef000c7a0 tx=0x7faef000cc60 comp rx=0 tx=0).stop 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 shutdown_connections 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef815ba50 0x7faef81537c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faed4067b20 0x7faed4069fe0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faef80ce320 0x7faef8165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 --2- 172.21.15.28:0/3576038598 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faef80cc170 0x7faef814c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.691+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 >> 172.21.15.28:0/3576038598 conn(0x7faef806fa90 msgr2=0x7faef80c81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:30.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.699+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 shutdown_connections 2024-03-21T08:42:30.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.699+0000 7faefe000640 1 -- 172.21.15.28:0/3576038598 wait complete. 2024-03-21T08:42:30.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 Processor -- start 2024-03-21T08:42:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 -- start start 2024-03-21T08:42:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14300ce490 0x7f143015e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f143015ba70 0x7f14300ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143015bf00 con 0x7f1430071980 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14300cedb0 con 0x7f14300ce490 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f1435400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143015ebf0 con 0x7f143015ba70 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14300ce490 0x7f143015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14300ce490 0x7f143015e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43010/0 (socket says 172.21.15.28:43010) 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 -- 172.21.15.28:0/771160291 learned_addr learned my addr 172.21.15.28:0/771160291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 -- 172.21.15.28:0/771160291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f143015ba70 msgr2=0x7f14300ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 --2- 172.21.15.28:0/771160291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f143015ba70 0x7f14300ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 -- 172.21.15.28:0/771160291 >> v1:172.21.15.28:6789/0 conn(0x7f1430071980 legacy=0x7f1430071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142e800640 1 -- 172.21.15.28:0/771160291 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143015eed0 con 0x7f14300ce490 2024-03-21T08:42:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.871+0000 7f142de00640 1 -- 172.21.15.28:0/771160291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917534687 0 0) 0x7f143015bf00 con 0x7f1430071980 2024-03-21T08:42:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f142e800640 1 --2- 172.21.15.28:0/771160291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143015e610 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb54a7685838116 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f142de00640 1 -- 172.21.15.28:0/771160291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14200b1070 con 0x7f14300ce490 2024-03-21T08:42:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f142de00640 1 -- 172.21.15.28:0/771160291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14200ac620 con 0x7f14300ce490 2024-03-21T08:42:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f142de00640 1 -- 172.21.15.28:0/771160291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14200ac9f0 con 0x7f14300ce490 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 -- 172.21.15.28:0/771160291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 msgr2=0x7f143015e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 --2- 172.21.15.28:0/771160291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143015e610 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142001c4d0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 -- 172.21.15.28:0/771160291 shutdown_connections 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 --2- 172.21.15.28:0/771160291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f143015ba70 0x7f14300ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 --2- 172.21.15.28:0/771160291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143015e610 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 -- 172.21.15.28:0/771160291 >> 172.21.15.28:0/771160291 conn(0x7f14300cc230 msgr2=0x7f14300cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 -- 172.21.15.28:0/771160291 shutdown_connections 2024-03-21T08:42:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.875+0000 7f1435400640 1 -- 172.21.15.28:0/771160291 wait complete. 2024-03-21T08:42:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 Processor -- start 2024-03-21T08:42:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- start start 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 0x7f1430153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 0x7f143006e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14301681c0 con 0x7f14300ce490 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1430155390 con 0x7f143015ba70 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1430168340 con 0x7f1430071980 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43016/0 (socket says 172.21.15.28:43016) 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 -- 172.21.15.28:0/2839405874 learned_addr learned my addr 172.21.15.28:0/2839405874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f142e800640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 0x7f1430153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f142f200640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 0x7f143006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 -- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 msgr2=0x7f1430153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 0x7f1430153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 -- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 msgr2=0x7f143006e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 0x7f143006e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 -- 172.21.15.28:0/2839405874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430136f50 con 0x7f14300ce490 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f142e800640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 0x7f1430153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f142f200640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 0x7f143006e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1425e00640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f141800b550 tx=0x7f141800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418014070 con 0x7f14300ce490 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1418010890 con 0x7f14300ce490 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418010c80 con 0x7f14300ce490 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143014f900 con 0x7f14300ce490 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.879+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14301354a0 con 0x7f14300ce490 2024-03-21T08:42:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143014fe20 con 0x7f14300ce490 2024-03-21T08:42:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1418017070 con 0x7f14300ce490 2024-03-21T08:42:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f1427400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 0x7f140c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1418085ef0 con 0x7f14300ce490 2024-03-21T08:42:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f142e800640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 0x7f140c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.883+0000 7f142e800640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 0x7f140c069fe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f14200ae000 tx=0x7f14200a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:30.887+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f143014fe20 con 0x7f14300ce490 2024-03-21T08:42:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.051+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f143004f410 con 0x7f14300ce490 2024-03-21T08:42:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.051+0000 7f1427400640 1 -- 172.21.15.28:0/2839405874 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f1418010e20 con 0x7f14300ce490 2024-03-21T08:42:31.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:31.055 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:30.068240+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 msgr2=0x7f140c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 0x7f140c069fe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f14200ae000 tx=0x7f14200a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 msgr2=0x7f143014ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f141800b550 tx=0x7f141800ba10 comp rx=0 tx=0).stop 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 shutdown_connections 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f143015ba70 0x7f143006e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14300ce490 0x7f143014ee60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f140c067b20 0x7f140c069fe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 --2- 172.21.15.28:0/2839405874 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1430071980 0x7f1430153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 >> 172.21.15.28:0/2839405874 conn(0x7f14300cc230 msgr2=0x7f14300b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 shutdown_connections 2024-03-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.055+0000 7f1435400640 1 -- 172.21.15.28:0/2839405874 wait complete. 2024-03-21T08:42:31.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:31.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 Processor -- start 2024-03-21T08:42:31.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 -- start start 2024-03-21T08:42:31.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e8415d060 0x7f6e8409af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e840b1360 0x7f6e840b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840d0780 con 0x7f6e84099b80 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840cc810 con 0x7f6e840b1360 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e8b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840ccaf0 con 0x7f6e8415d060 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e7be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e840b1360 0x7f6e840b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e89400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6e84099b80 0x7f6e840cd2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54464/0 (socket says 172.21.15.28:54464) 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e89400640 1 -- 172.21.15.28:0/157234814 learned_addr learned my addr 172.21.15.28:0/157234814 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e88a00640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e8415d060 0x7f6e8409af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e7be00640 1 -- 172.21.15.28:0/157234814 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e8415d060 msgr2=0x7f6e8409af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e7be00640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e8415d060 0x7f6e8409af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e83e00640 1 -- 172.21.15.28:0/157234814 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974181600 0 0) 0x7f6e840d0780 con 0x7f6e84099b80 2024-03-21T08:42:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e7be00640 1 -- 172.21.15.28:0/157234814 >> v1:172.21.15.28:6789/0 conn(0x7f6e84099b80 legacy=0x7f6e840cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.215+0000 7f6e7be00640 1 -- 172.21.15.28:0/157234814 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840ccdd0 con 0x7f6e840b1360 2024-03-21T08:42:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e7be00640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e840b1360 0x7f6e840b1780 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0097c0 tx=0x7f6e6c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ed4bbc6c5abcd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e83e00640 1 -- 172.21.15.28:0/157234814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0aa620 con 0x7f6e840b1360 2024-03-21T08:42:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e83e00640 1 -- 172.21.15.28:0/157234814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c0af070 con 0x7f6e840b1360 2024-03-21T08:42:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e83e00640 1 -- 172.21.15.28:0/157234814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0aa9e0 con 0x7f6e840b1360 2024-03-21T08:42:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 -- 172.21.15.28:0/157234814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e840b1360 msgr2=0x7f6e840b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e840b1360 0x7f6e840b1780 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0097c0 tx=0x7f6e6c01c460 comp rx=0 tx=0).stop 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 -- 172.21.15.28:0/157234814 shutdown_connections 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e840b1360 0x7f6e840b1780 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 --2- 172.21.15.28:0/157234814 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e8415d060 0x7f6e8409af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 -- 172.21.15.28:0/157234814 >> 172.21.15.28:0/157234814 conn(0x7f6e840b3510 msgr2=0x7f6e840b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 -- 172.21.15.28:0/157234814 shutdown_connections 2024-03-21T08:42:31.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.219+0000 7f6e8b400640 1 -- 172.21.15.28:0/157234814 wait complete. 2024-03-21T08:42:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 Processor -- start 2024-03-21T08:42:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- start start 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 0x7f6e841f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e840b1360 0x7f6e8406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840b0b70 con 0x7f6e8415d060 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840b09f0 con 0x7f6e840b1360 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840d0310 con 0x7f6e84099b80 2024-03-21T08:42:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43034/0 (socket says 172.21.15.28:43034) 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 -- 172.21.15.28:0/1678334663 learned_addr learned my addr 172.21.15.28:0/1678334663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e88a00640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 0x7f6e841f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 -- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 msgr2=0x7f6e841f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 0x7f6e841f6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 -- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e840b1360 msgr2=0x7f6e8406d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e840b1360 0x7f6e8406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 -- 172.21.15.28:0/1678334663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84163ee0 con 0x7f6e8415d060 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e88a00640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 0x7f6e841f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e89400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c7a0 tx=0x7f6e7c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c018070 con 0x7f6e8415d060 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c00a640 con 0x7f6e8415d060 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00aab0 con 0x7f6e8415d060 2024-03-21T08:42:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e841641c0 con 0x7f6e8415d060 2024-03-21T08:42:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.223+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e84164780 con 0x7f6e8415d060 2024-03-21T08:42:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8404f410 con 0x7f6e8415d060 2024-03-21T08:42:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6e7c01b070 con 0x7f6e8415d060 2024-03-21T08:42:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e81600640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 0x7f6e60069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e7c0856b0 con 0x7f6e8415d060 2024-03-21T08:42:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e88a00640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 0x7f6e60069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.227+0000 7f6e88a00640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 0x7f6e60069fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6e740097a0 tx=0x7f6e74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.231+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6e7c055f30 con 0x7f6e8415d060 2024-03-21T08:42:31.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e840b41b0 con 0x7f6e8415d060 2024-03-21T08:42:31.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e81600640 1 -- 172.21.15.28:0/1678334663 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f6e7c055830 con 0x7f6e8415d060 2024-03-21T08:42:31.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:31.398 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:30.068240+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 msgr2=0x7f6e60069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 0x7f6e60069fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6e740097a0 tx=0x7f6e74008040 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 msgr2=0x7f6e8406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c7a0 tx=0x7f6e7c00cc60 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 shutdown_connections 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e8415d060 0x7f6e8406dd20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e840b1360 0x7f6e8406d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e60067b20 0x7f6e60069fe0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 --2- 172.21.15.28:0/1678334663 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e84099b80 0x7f6e841f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.395+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 >> 172.21.15.28:0/1678334663 conn(0x7f6e840b3510 msgr2=0x7f6e8415b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.399+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 shutdown_connections 2024-03-21T08:42:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.399+0000 7f6e8b400640 1 -- 172.21.15.28:0/1678334663 wait complete. 2024-03-21T08:42:31.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-21T08:42:31.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 Processor -- start 2024-03-21T08:42:31.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 -- start start 2024-03-21T08:42:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0a40b4420 0x7fd0a4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd0a40b4800 0x7fd0a4071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a415bf90 con 0x7fd0a40ccbf0 2024-03-21T08:42:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a415b1d0 con 0x7fd0a40b4420 2024-03-21T08:42:31.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a9e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a415b4b0 con 0x7fd0a40b4800 2024-03-21T08:42:31.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0a40b4420 0x7fd0a4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd0a40ccbf0 0x7fd0a40cd0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54478/0 (socket says 172.21.15.28:54478) 2024-03-21T08:42:31.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a3400640 1 -- 172.21.15.28:0/4274334735 learned_addr learned my addr 172.21.15.28:0/4274334735 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:31.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd09be00640 1 --2- 172.21.15.28:0/4274334735 >> v2:172.21.15.151:3300/0 conn(0x7fd0a40b4800 0x7fd0a4071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2000640 1 -- 172.21.15.28:0/4274334735 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474483852 0 0) 0x7fd0a415bf90 con 0x7fd0a40ccbf0 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2000640 1 -- 172.21.15.28:0/4274334735 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd088003530 con 0x7fd0a40ccbf0 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2a00640 1 -- 172.21.15.28:0/4274334735 >> v2:172.21.15.151:3300/0 conn(0x7fd0a40b4800 msgr2=0x7fd0a4071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/4274334735 >> v2:172.21.15.151:3300/0 conn(0x7fd0a40b4800 0x7fd0a4071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2a00640 1 -- 172.21.15.28:0/4274334735 >> v1:172.21.15.28:6789/0 conn(0x7fd0a40ccbf0 legacy=0x7fd0a40cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.563+0000 7fd0a2a00640 1 -- 172.21.15.28:0/4274334735 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a415b790 con 0x7fd0a40b4420 2024-03-21T08:42:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/4274334735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a40b4420 0x7fd0a4071a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40cc6f0 tx=0x7fd09401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e755ba5d3d29354f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a2000640 1 -- 172.21.15.28:0/4274334735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0940b1070 con 0x7fd0a40b4420 2024-03-21T08:42:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a2000640 1 -- 172.21.15.28:0/4274334735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0940a8050 con 0x7fd0a40b4420 2024-03-21T08:42:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a2000640 1 -- 172.21.15.28:0/4274334735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0940ac840 con 0x7fd0a40b4420 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- 172.21.15.28:0/4274334735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a40b4420 msgr2=0x7fd0a4071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/4274334735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a40b4420 0x7fd0a4071a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40cc6f0 tx=0x7fd09401c450 comp rx=0 tx=0).stop 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- 172.21.15.28:0/4274334735 shutdown_connections 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/4274334735 >> v2:172.21.15.151:3300/0 conn(0x7fd0a40b4800 0x7fd0a4071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/4274334735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a40b4420 0x7fd0a4071a80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- 172.21.15.28:0/4274334735 >> 172.21.15.28:0/4274334735 conn(0x7fd0a40674f0 msgr2=0x7fd0a406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- 172.21.15.28:0/4274334735 shutdown_connections 2024-03-21T08:42:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- 172.21.15.28:0/4274334735 wait complete. 2024-03-21T08:42:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 Processor -- start 2024-03-21T08:42:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.567+0000 7fd0a9e00640 1 -- start start 2024-03-21T08:42:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 0x7fd0a413a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 0x7fd0a4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40b36b0 con 0x7fd0a41564d0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40b3530 con 0x7fd0a40b4420 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40b3830 con 0x7fd0a40ccbf0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43064/0 (socket says 172.21.15.28:43064) 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 -- 172.21.15.28:0/686893510 learned_addr learned my addr 172.21.15.28:0/686893510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 -- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 msgr2=0x7fd0a4155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd09be00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 0x7fd0a4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 0x7fd0a413a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 0x7fd0a4155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 -- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 msgr2=0x7fd0a413a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 0x7fd0a413a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 -- 172.21.15.28:0/686893510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a41569c0 con 0x7fd0a41564d0 2024-03-21T08:42:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd09be00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 0x7fd0a4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 0x7fd0a413a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a3400640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c9f0 tx=0x7fd09c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c017070 con 0x7fd0a41564d0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c00ab70 con 0x7fd0a41564d0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a4156ca0 con 0x7fd0a41564d0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c012400 con 0x7fd0a41564d0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a41f4cf0 con 0x7fd0a41564d0 2024-03-21T08:42:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.571+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a4165580 con 0x7fd0a41564d0 2024-03-21T08:42:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.575+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd09c01a070 con 0x7fd0a41564d0 2024-03-21T08:42:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.575+0000 7fd09b400640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 0x7fd08006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.575+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd09c085c40 con 0x7fd0a41564d0 2024-03-21T08:42:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.575+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 0x7fd08006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.575+0000 7fd0a2a00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 0x7fd08006a010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd0940ae000 tx=0x7fd094008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.579+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd09c055e70 con 0x7fd0a41564d0 2024-03-21T08:42:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:31.779+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd0a404f4f0 con 0x7fd0a41564d0 2024-03-21T08:42:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd09b400640 1 -- 172.21.15.28:0/686893510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd09c059f10 con 0x7fd0a41564d0 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 msgr2=0x7fd08006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 0x7fd08006a010 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd0940ae000 tx=0x7fd094008040 comp rx=0 tx=0).stop 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 msgr2=0x7fd0a4157820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c9f0 tx=0x7fd09c00ceb0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 shutdown_connections 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a41564d0 0x7fd0a4157820 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd080067b50 0x7fd08006a010 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0a40ccbf0 0x7fd0a4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 --2- 172.21.15.28:0/686893510 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0a40b4420 0x7fd0a413a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 >> 172.21.15.28:0/686893510 conn(0x7fd0a40674f0 msgr2=0x7fd0a414f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 shutdown_connections 2024-03-21T08:42:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.687+0000 7fd0a9e00640 1 -- 172.21.15.28:0/686893510 wait complete. 2024-03-21T08:42:32.710 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-21T08:42:32.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-21T08:42:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 Processor -- start 2024-03-21T08:42:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 -- start start 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff80b4420 0x7faff8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faff80b4800 0x7faff8071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faff80ccbf0 0x7faff80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff815bf90 con 0x7faff80ccbf0 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff815b1d0 con 0x7faff80b4800 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faffd200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff815b4b0 con 0x7faff80b4420 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faff80ccbf0 0x7faff80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faff80ccbf0 0x7faff80cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43080/0 (socket says 172.21.15.28:43080) 2024-03-21T08:42:32.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 -- 172.21.15.28:0/1371792966 learned_addr learned my addr 172.21.15.28:0/1371792966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7fafefe00640 1 --2- 172.21.15.28:0/1371792966 >> v2:172.21.15.151:3300/0 conn(0x7faff80b4800 0x7faff8071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff5e00640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff80b4420 0x7faff8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 -- 172.21.15.28:0/1371792966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff80b4420 msgr2=0x7faff8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff80b4420 0x7faff8071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 -- 172.21.15.28:0/1371792966 >> v2:172.21.15.151:3300/0 conn(0x7faff80b4800 msgr2=0x7faff8071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 --2- 172.21.15.28:0/1371792966 >> v2:172.21.15.151:3300/0 conn(0x7faff80b4800 0x7faff8071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.855+0000 7faff6800640 1 -- 172.21.15.28:0/1371792966 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff815b790 con 0x7faff80ccbf0 2024-03-21T08:42:32.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.859+0000 7faff6800640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff80cd0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faff806fca0 tx=0x7faff001ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4c8108522559c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.859+0000 7faff5400640 1 -- 172.21.15.28:0/1371792966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff000b2d0 con 0x7faff80ccbf0 2024-03-21T08:42:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.859+0000 7faff5400640 1 -- 172.21.15.28:0/1371792966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff000b470 con 0x7faff80ccbf0 2024-03-21T08:42:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.859+0000 7faff5400640 1 -- 172.21.15.28:0/1371792966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff00a7c90 con 0x7faff80ccbf0 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- 172.21.15.28:0/1371792966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 msgr2=0x7faff80cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff80cd0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faff806fca0 tx=0x7faff001ebc0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- 172.21.15.28:0/1371792966 shutdown_connections 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff80cd0a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- 172.21.15.28:0/1371792966 >> v2:172.21.15.151:3300/0 conn(0x7faff80b4800 0x7faff8071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- 172.21.15.28:0/1371792966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff80b4420 0x7faff8071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- 172.21.15.28:0/1371792966 >> 172.21.15.28:0/1371792966 conn(0x7faff80674f0 msgr2=0x7faff806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- 172.21.15.28:0/1371792966 shutdown_connections 2024-03-21T08:42:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- 172.21.15.28:0/1371792966 wait complete. 2024-03-21T08:42:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 Processor -- start 2024-03-21T08:42:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- start start 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff80b4420 0x7faff81497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff8164480 0x7faff80c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff815efb0 con 0x7faff80ccbf0 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff815ee30 con 0x7faff80b4420 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faffd200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff815f130 con 0x7faff8164480 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43092/0 (socket says 172.21.15.28:43092) 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 -- 172.21.15.28:0/4288853667 learned_addr learned my addr 172.21.15.28:0/4288853667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faff5e00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff80b4420 0x7faff81497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7faff6800640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff8164480 0x7faff80c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 -- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff8164480 msgr2=0x7faff80c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff8164480 0x7faff80c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 -- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff80b4420 msgr2=0x7faff81497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff80b4420 0x7faff81497b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.863+0000 7fafefe00640 1 -- 172.21.15.28:0/4288853667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff81f9a30 con 0x7faff80ccbf0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafefe00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fafe000e900 tx=0x7fafe000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0017070 con 0x7faff80ccbf0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe000ca00 con 0x7faff80ccbf0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe000cdd0 con 0x7faff80ccbf0 2024-03-21T08:42:32.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff81f9d10 con 0x7faff80ccbf0 2024-03-21T08:42:32.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff81fa2d0 con 0x7faff80ccbf0 2024-03-21T08:42:32.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff8164890 con 0x7faff80ccbf0 2024-03-21T08:42:32.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fafe001a070 con 0x7faff80ccbf0 2024-03-21T08:42:32.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 0x7fafd4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:32.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fafe0086310 con 0x7faff80ccbf0 2024-03-21T08:42:32.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.867+0000 7faff5e00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 0x7fafd4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.871+0000 7faff5e00640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 0x7fafd4069fe0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009b60 tx=0x7fafe8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:32.875+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faff8164890 con 0x7faff80ccbf0 2024-03-21T08:42:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.071+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7faff8073430 con 0x7faff80ccbf0 2024-03-21T08:42:33.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.691+0000 7fafee000640 1 -- 172.21.15.28:0/4288853667 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafe0056530 con 0x7faff80ccbf0 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 msgr2=0x7fafd4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 0x7fafd4069fe0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009b60 tx=0x7fafe8009190 comp rx=0 tx=0).stop 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 msgr2=0x7faff8163f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fafe000e900 tx=0x7fafe000edc0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 shutdown_connections 2024-03-21T08:42:33.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fafd4067b20 0x7fafd4069fe0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faff8164480 0x7faff80c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faff80ccbf0 0x7faff8163f40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 --2- 172.21.15.28:0/4288853667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faff80b4420 0x7faff81497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 >> 172.21.15.28:0/4288853667 conn(0x7faff80674f0 msgr2=0x7faff8149ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 shutdown_connections 2024-03-21T08:42:33.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.695+0000 7faffd200640 1 -- 172.21.15.28:0/4288853667 wait complete. 2024-03-21T08:42:33.717 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-21T08:42:33.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 Processor -- start 2024-03-21T08:42:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 -- start start 2024-03-21T08:42:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 0x7f4b8c15bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0cf5b0 con 0x7f4b8c15abf0 2024-03-21T08:42:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c15afd0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.855+0000 7f4b91400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c15c0d0 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b8a800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4b8c15abf0 0x7f4b8c0d07a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:45370/0 (socket says 172.21.15.28:45370) 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b8b200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 0x7f4b8c15bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b8a800640 1 -- 172.21.15.28:0/3101477323 learned_addr learned my addr 172.21.15.28:0/3101477323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b89e00640 1 -- 172.21.15.28:0/3101477323 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866338632 0 0) 0x7f4b8c0cf5b0 con 0x7f4b8c15abf0 2024-03-21T08:42:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b89e00640 1 -- 172.21.15.28:0/3101477323 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b70003530 con 0x7f4b8c15abf0 2024-03-21T08:42:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b83e00640 1 -- 172.21.15.28:0/3101477323 >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 msgr2=0x7f4b8c15bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b83e00640 1 --2- 172.21.15.28:0/3101477323 >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 0x7f4b8c15bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b83e00640 1 -- 172.21.15.28:0/3101477323 >> v1:172.21.15.151:6789/0 conn(0x7f4b8c15abf0 legacy=0x7f4b8c0d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b83e00640 1 -- 172.21.15.28:0/3101477323 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c15e8d0 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.859+0000 7f4b8b200640 1 --2- 172.21.15.28:0/3101477323 >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 0x7f4b8c15bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b83e00640 1 --2- 172.21.15.28:0/3101477323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0ceed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b740097c0 tx=0x7f4b7401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d825feeeec736c4a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b89e00640 1 -- 172.21.15.28:0/3101477323 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b740b0070 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b89e00640 1 -- 172.21.15.28:0/3101477323 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b740ab600 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b89e00640 1 -- 172.21.15.28:0/3101477323 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b740ab9b0 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 -- 172.21.15.28:0/3101477323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 msgr2=0x7f4b8c0ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 --2- 172.21.15.28:0/3101477323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0ceed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b740097c0 tx=0x7f4b7401c380 comp rx=0 tx=0).stop 2024-03-21T08:42:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 -- 172.21.15.28:0/3101477323 shutdown_connections 2024-03-21T08:42:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 --2- 172.21.15.28:0/3101477323 >> v2:172.21.15.151:3300/0 conn(0x7f4b8c0d0eb0 0x7f4b8c15bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 --2- 172.21.15.28:0/3101477323 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0ceed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 -- 172.21.15.28:0/3101477323 >> 172.21.15.28:0/3101477323 conn(0x7f4b8c099a30 msgr2=0x7f4b8c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 -- 172.21.15.28:0/3101477323 shutdown_connections 2024-03-21T08:42:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.863+0000 7f4b91400640 1 -- 172.21.15.28:0/3101477323 wait complete. 2024-03-21T08:42:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 Processor -- start 2024-03-21T08:42:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- start start 2024-03-21T08:42:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b8c15abf0 0x7f4b8c0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0ccfd0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0cce50 con 0x7f4b8c15abf0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c15ef20 con 0x7f4b8c15f3b0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43096/0 (socket says 172.21.15.28:43096) 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 -- 172.21.15.28:0/1892280050 learned_addr learned my addr 172.21.15.28:0/1892280050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8b200640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b83e00640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b8c15abf0 0x7f4b8c0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 -- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 msgr2=0x7f4b8c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 -- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b8c15abf0 msgr2=0x7f4b8c0c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b8c15abf0 0x7f4b8c0c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 -- 172.21.15.28:0/1892280050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0c5180 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b8a800640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c3c0 tx=0x7f4b7c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c017070 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c005db0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0072f0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c0c5420 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.867+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c0c56c0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c04f410 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4b7c01a070 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b82a00640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 0x7f4b68069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4b7c0854c0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b83e00640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 0x7f4b68069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.871+0000 7f4b83e00640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 0x7f4b68069fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4b740ad000 tx=0x7f4b740a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:33.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:33.875+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4b7c0556e0 con 0x7f4b8c0d0eb0 2024-03-21T08:42:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.071+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b8c152220 con 0x7f4b8c0d0eb0 2024-03-21T08:42:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.071+0000 7f4b82a00640 1 -- 172.21.15.28:0/1892280050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f4b7c059780 con 0x7f4b8c0d0eb0 2024-03-21T08:42:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:34.078 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:33.074369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 msgr2=0x7f4b68069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 0x7f4b68069fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4b740ad000 tx=0x7f4b740a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 msgr2=0x7f4b8c0c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c3c0 tx=0x7f4b7c00c7c0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 shutdown_connections 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4b68067b20 0x7f4b68069fe0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4b8c15f3b0 0x7f4b8c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4b8c15abf0 0x7f4b8c0c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 --2- 172.21.15.28:0/1892280050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c0d0eb0 0x7f4b8c0c61a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 >> 172.21.15.28:0/1892280050 conn(0x7f4b8c099a30 msgr2=0x7f4b8c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 shutdown_connections 2024-03-21T08:42:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.075+0000 7f4b91400640 1 -- 172.21.15.28:0/1892280050 wait complete. 2024-03-21T08:42:34.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 Processor -- start 2024-03-21T08:42:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 -- start start 2024-03-21T08:42:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f19d806c180 0x7f19d806c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19d815d7a0 0x7f19d80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d80b4950 con 0x7f19d806e8e0 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d815c300 con 0x7f19d815d7a0 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d806caa0 con 0x7f19d806c180 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19d815d7a0 0x7f19d80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19d815d7a0 0x7f19d80b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43106/0 (socket says 172.21.15.28:43106) 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 -- 172.21.15.28:0/820813907 learned_addr learned my addr 172.21.15.28:0/820813907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19cfe00640 1 --1- 172.21.15.28:0/820813907 >> v1:172.21.15.28:6789/0 conn(0x7f19d806e8e0 0x7f19d80b2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54488/0 (socket says 172.21.15.28:54488) 2024-03-21T08:42:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d6a00640 1 --2- 172.21.15.28:0/820813907 >> v2:172.21.15.151:3300/0 conn(0x7f19d806c180 0x7f19d806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 -- 172.21.15.28:0/820813907 >> v2:172.21.15.151:3300/0 conn(0x7f19d806c180 msgr2=0x7f19d806c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 --2- 172.21.15.28:0/820813907 >> v2:172.21.15.151:3300/0 conn(0x7f19d806c180 0x7f19d806c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 -- 172.21.15.28:0/820813907 >> v1:172.21.15.28:6789/0 conn(0x7f19d806e8e0 legacy=0x7f19d80b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d7400640 1 -- 172.21.15.28:0/820813907 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80b4bf0 con 0x7f19d815d7a0 2024-03-21T08:42:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.239+0000 7f19d6000640 1 -- 172.21.15.28:0/820813907 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473005411 0 0) 0x7f19d80b4950 con 0x7f19d806e8e0 2024-03-21T08:42:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7400640 1 --2- 172.21.15.28:0/820813907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d815d7a0 0x7f19d80b3510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19d80717b0 tx=0x7f19d001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341327ce5ae8e5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d6000640 1 -- 172.21.15.28:0/820813907 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00a5620 con 0x7f19d815d7a0 2024-03-21T08:42:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d6000640 1 -- 172.21.15.28:0/820813907 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d00a57c0 con 0x7f19d815d7a0 2024-03-21T08:42:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d6000640 1 -- 172.21.15.28:0/820813907 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00a5b90 con 0x7f19d815d7a0 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 -- 172.21.15.28:0/820813907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d815d7a0 msgr2=0x7f19d80b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 --2- 172.21.15.28:0/820813907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d815d7a0 0x7f19d80b3510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19d80717b0 tx=0x7f19d001c960 comp rx=0 tx=0).stop 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 -- 172.21.15.28:0/820813907 shutdown_connections 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 --2- 172.21.15.28:0/820813907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d815d7a0 0x7f19d80b3510 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 --2- 172.21.15.28:0/820813907 >> v2:172.21.15.151:3300/0 conn(0x7f19d806c180 0x7f19d806c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 -- 172.21.15.28:0/820813907 >> 172.21.15.28:0/820813907 conn(0x7f19d8072080 msgr2=0x7f19d8072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 -- 172.21.15.28:0/820813907 shutdown_connections 2024-03-21T08:42:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.243+0000 7f19d7e00640 1 -- 172.21.15.28:0/820813907 wait complete. 2024-03-21T08:42:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 Processor -- start 2024-03-21T08:42:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- start start 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d806c180 0x7f19d814fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 0x7f19d81488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80b3bd0 con 0x7f19d806e8e0 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80b3a50 con 0x7f19d806c180 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8168e30 con 0x7f19d815d7a0 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43120/0 (socket says 172.21.15.28:43120) 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 -- 172.21.15.28:0/846286961 learned_addr learned my addr 172.21.15.28:0/846286961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7400640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 0x7f19d81488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d6a00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d806c180 0x7f19d814fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 -- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 msgr2=0x7f19d81488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 0x7f19d81488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 -- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d806c180 msgr2=0x7f19d814fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d806c180 0x7f19d814fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 -- 172.21.15.28:0/846286961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80add50 con 0x7f19d806e8e0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7400640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 0x7f19d81488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cfe00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c520 tx=0x7f19c000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0018070 con 0x7f19d806e8e0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c00074e0 con 0x7f19d806e8e0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c00049f0 con 0x7f19d806e8e0 2024-03-21T08:42:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d8151650 con 0x7f19d806e8e0 2024-03-21T08:42:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d8151bd0 con 0x7f19d806e8e0 2024-03-21T08:42:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.247+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8075270 con 0x7f19d806e8e0 2024-03-21T08:42:34.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.251+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f19c001b070 con 0x7f19d806e8e0 2024-03-21T08:42:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.251+0000 7f19cea00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 0x7f19b4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.251+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f19c0085520 con 0x7f19d806e8e0 2024-03-21T08:42:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.251+0000 7f19d6a00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 0x7f19b4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.255+0000 7f19d6a00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 0x7f19b4069fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b930 tx=0x7f19c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.255+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f19c0055d90 con 0x7f19d806e8e0 2024-03-21T08:42:34.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.419+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d815ea60 con 0x7f19d806e8e0 2024-03-21T08:42:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19cea00640 1 -- 172.21.15.28:0/846286961 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16954 (secure 0 0 0) 0x7f19c0055690 con 0x7f19d806e8e0 2024-03-21T08:42:34.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:34.425 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:31.679183+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 msgr2=0x7f19b4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 0x7f19b4069fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b930 tx=0x7f19c8008040 comp rx=0 tx=0).stop 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 msgr2=0x7f19d80ad810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c520 tx=0x7f19c000c9e0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 shutdown_connections 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f19b4067b20 0x7f19b4069fe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f19d815d7a0 0x7f19d81488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19d806e8e0 0x7f19d80ad810 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 --2- 172.21.15.28:0/846286961 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f19d806c180 0x7f19d814fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 >> 172.21.15.28:0/846286961 conn(0x7f19d8072080 msgr2=0x7f19d815adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 shutdown_connections 2024-03-21T08:42:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.423+0000 7f19d7e00640 1 -- 172.21.15.28:0/846286961 wait complete. 2024-03-21T08:42:34.446 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-21T08:42:34.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:42:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.583+0000 7f3a6aa00640 1 Processor -- start 2024-03-21T08:42:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6aa00640 1 -- start start 2024-03-21T08:42:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a6406c490 0x7f3a640cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640d0380 con 0x7f3a6409a140 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6aa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640d0660 con 0x7f3a640b0ae0 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a640d0150 con 0x7f3a6406c490 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a6406c490 0x7f3a640cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a68c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3a6409a140 0x7f3a6415d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54496/0 (socket says 172.21.15.28:54496) 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a68c00640 1 -- 172.21.15.28:0/3165694254 learned_addr learned my addr 172.21.15.28:0/3165694254 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6a000640 1 -- 172.21.15.28:0/3165694254 >> v1:172.21.15.151:6789/0 conn(0x7f3a640b0ae0 legacy=0x7f3a64099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6a000640 1 -- 172.21.15.28:0/3165694254 >> v1:172.21.15.28:6789/0 conn(0x7f3a6409a140 legacy=0x7f3a6415d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a6a000640 1 -- 172.21.15.28:0/3165694254 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640d0940 con 0x7f3a6406c490 2024-03-21T08:42:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.587+0000 7f3a63e00640 1 -- 172.21.15.28:0/3165694254 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999235360 0 0) 0x7f3a640d0380 con 0x7f3a6409a140 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6a000640 1 --2- 172.21.15.28:0/3165694254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6406c490 0x7f3a640cfc10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009d90 tx=0x7f3a5c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3ec34d14183b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a63e00640 1 -- 172.21.15.28:0/3165694254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0b2070 con 0x7f3a6406c490 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a63e00640 1 -- 172.21.15.28:0/3165694254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c0a5760 con 0x7f3a6406c490 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a63e00640 1 -- 172.21.15.28:0/3165694254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0a5b30 con 0x7f3a6406c490 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3165694254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6406c490 msgr2=0x7f3a640cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3165694254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6406c490 0x7f3a640cfc10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009d90 tx=0x7f3a5c01c8a0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3165694254 shutdown_connections 2024-03-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3165694254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6406c490 0x7f3a640cfc10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3165694254 >> 172.21.15.28:0/3165694254 conn(0x7f3a640b3390 msgr2=0x7f3a640b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3165694254 shutdown_connections 2024-03-21T08:42:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.591+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3165694254 wait complete. 2024-03-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 Processor -- start 2024-03-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- start start 2024-03-21T08:42:34.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 0x7f3a6414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a640b0ae0 0x7f3a640b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640cd080 con 0x7f3a6409a140 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a640ccf00 con 0x7f3a6406c490 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6415e030 con 0x7f3a640b0ae0 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43150/0 (socket says 172.21.15.28:43150) 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 -- 172.21.15.28:0/3072094592 learned_addr learned my addr 172.21.15.28:0/3072094592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6a000640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a640b0ae0 0x7f3a640b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a69600640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 0x7f3a6414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 -- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a640b0ae0 msgr2=0x7f3a640b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a640b0ae0 0x7f3a640b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 -- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 msgr2=0x7f3a6414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 0x7f3a6414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 -- 172.21.15.28:0/3072094592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640b4630 con 0x7f3a6409a140 2024-03-21T08:42:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a69600640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 0x7f3a6414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a68c00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000c4b0 tx=0x7f3a5000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50018070 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50007500 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a640b4910 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004a30 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.595+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a640b4ed0 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.599+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a64075270 con 0x7f3a6409a140 2024-03-21T08:42:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.599+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3a5001b070 con 0x7f3a6409a140 2024-03-21T08:42:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.599+0000 7f3a61600640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 0x7f3a44069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.599+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3a500852e0 con 0x7f3a6409a140 2024-03-21T08:42:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.599+0000 7f3a69600640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 0x7f3a44069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.603+0000 7f3a69600640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 0x7f3a44069fe0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a640d0120 tx=0x7f3a58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.603+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3a50055500 con 0x7f3a6409a140 2024-03-21T08:42:34.654 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-03-21T08:42:34.651+0000 7fdca0e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:42:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.775+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a64099e90 con 0x7f3a44067b20 2024-03-21T08:42:34.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.779+0000 7f3a61600640 1 -- 172.21.15.28:0/3072094592 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+243214 (secure 0 0 0) 0x7f3a64099e90 con 0x7f3a44067b20 2024-03-21T08:42:34.786 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:34.787 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:42:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 msgr2=0x7f3a44069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 0x7f3a44069fe0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3a640d0120 tx=0x7f3a58009040 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 msgr2=0x7f3a640c7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000c4b0 tx=0x7f3a5000c970 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 shutdown_connections 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3a44067b20 0x7f3a44069fe0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3a640b0ae0 0x7f3a640b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a6409a140 0x7f3a640c7f10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 --2- 172.21.15.28:0/3072094592 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3a6406c490 0x7f3a6414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 >> 172.21.15.28:0/3072094592 conn(0x7f3a640b3390 msgr2=0x7f3a64070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 shutdown_connections 2024-03-21T08:42:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.783+0000 7f3a6aa00640 1 -- 172.21.15.28:0/3072094592 wait complete. 2024-03-21T08:42:34.807 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-03-21T08:42:33.736599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232984,"kb_used_data":4316,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741736,"statfs":{"total":773094113280,"available":772855537664,"internally_reserved":0,"allocated":4419584,"data_stored":2547403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.076102"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671004+0000","last_change":"2024-03-21T08:42:29.671004+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671004+0000","last_undegraded":"2024-03-21T08:42:29.671004+0000","last_fullsized":"2024-03-21T08:42:29.671004+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670994+0000","last_change":"2024-03-21T08:42:29.670994+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670994+0000","last_undegraded":"2024-03-21T08:42:29.670994+0000","last_fullsized":"2024-03-21T08:42:29.670994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671282+0000","last_change":"2024-03-21T08:42:29.671282+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671282+0000","last_undegraded":"2024-03-21T08:42:29.671282+0000","last_fullsized":"2024-03-21T08:42:29.671282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670924+0000","last_change":"2024-03-21T08:42:29.670924+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670924+0000","last_undegraded":"2024-03-21T08:42:29.670924+0000","last_fullsized":"2024-03-21T08:42:29.670924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672336+0000","last_change":"2024-03-21T08:42:29.672336+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672336+0000","last_undegraded":"2024-03-21T08:42:29.672336+0000","last_fullsized":"2024-03-21T08:42:29.672336+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673666+0000","last_change":"2024-03-21T08:42:29.673666+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673666+0000","last_undegraded":"2024-03-21T08:42:29.673666+0000","last_fullsized":"2024-03-21T08:42:29.673666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670674+0000","last_change":"2024-03-21T08:42:29.670674+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670674+0000","last_undegraded":"2024-03-21T08:42:29.670674+0000","last_fullsized":"2024-03-21T08:42:29.670674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673436+0000","last_change":"2024-03-21T08:42:29.673436+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673436+0000","last_undegraded":"2024-03-21T08:42:29.673436+0000","last_fullsized":"2024-03-21T08:42:29.673436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677791+0000","last_change":"2024-03-21T08:42:29.677791+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677791+0000","last_undegraded":"2024-03-21T08:42:29.677791+0000","last_fullsized":"2024-03-21T08:42:29.677791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670239+0000","last_change":"2024-03-21T08:42:29.670239+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670239+0000","last_undegraded":"2024-03-21T08:42:29.670239+0000","last_fullsized":"2024-03-21T08:42:29.670239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671865+0000","last_change":"2024-03-21T08:42:29.671865+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671865+0000","last_undegraded":"2024-03-21T08:42:29.671865+0000","last_fullsized":"2024-03-21T08:42:29.671865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672819+0000","last_change":"2024-03-21T08:42:29.672819+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672819+0000","last_undegraded":"2024-03-21T08:42:29.672819+0000","last_fullsized":"2024-03-21T08:42:29.672819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671261+0000","last_change":"2024-03-21T08:42:29.671261+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671261+0000","last_undegraded":"2024-03-21T08:42:29.671261+0000","last_fullsized":"2024-03-21T08:42:29.671261+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672622+0000","last_change":"2024-03-21T08:42:29.672622+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672622+0000","last_undegraded":"2024-03-21T08:42:29.672622+0000","last_fullsized":"2024-03-21T08:42:29.672622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671020+0000","last_change":"2024-03-21T08:42:29.671020+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671020+0000","last_undegraded":"2024-03-21T08:42:29.671020+0000","last_fullsized":"2024-03-21T08:42:29.671020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668449+0000","last_change":"2024-03-21T08:42:29.668449+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668449+0000","last_undegraded":"2024-03-21T08:42:29.668449+0000","last_fullsized":"2024-03-21T08:42:29.668449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671972+0000","last_change":"2024-03-21T08:42:29.671972+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671972+0000","last_undegraded":"2024-03-21T08:42:29.671972+0000","last_fullsized":"2024-03-21T08:42:29.671972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668747+0000","last_change":"2024-03-21T08:42:29.668747+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668747+0000","last_undegraded":"2024-03-21T08:42:29.668747+0000","last_fullsized":"2024-03-21T08:42:29.668747+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688377+0000","last_change":"2024-03-21T08:42:29.683944+0000","last_active":"2024-03-21T08:42:29.688377+0000","last_peered":"2024-03-21T08:42:29.688377+0000","last_clean":"2024-03-21T08:42:29.688377+0000","last_became_active":"2024-03-21T08:42:29.683089+0000","last_became_peered":"2024-03-21T08:42:29.683089+0000","last_unstale":"2024-03-21T08:42:29.688377+0000","last_undegraded":"2024-03-21T08:42:29.688377+0000","last_fullsized":"2024-03-21T08:42:29.688377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:21:01.343294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672583+0000","last_change":"2024-03-21T08:42:29.672583+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672583+0000","last_undegraded":"2024-03-21T08:42:29.672583+0000","last_fullsized":"2024-03-21T08:42:29.672583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667838+0000","last_change":"2024-03-21T08:42:28.667838+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667838+0000","last_undegraded":"2024-03-21T08:42:28.667838+0000","last_fullsized":"2024-03-21T08:42:28.667838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.686362+0000","last_change":"2024-03-21T08:42:29.683368+0000","last_active":"2024-03-21T08:42:29.686362+0000","last_peered":"2024-03-21T08:42:29.686362+0000","last_clean":"2024-03-21T08:42:29.686362+0000","last_became_active":"2024-03-21T08:42:29.683095+0000","last_became_peered":"2024-03-21T08:42:29.683095+0000","last_unstale":"2024-03-21T08:42:29.686362+0000","last_undegraded":"2024-03-21T08:42:29.686362+0000","last_fullsized":"2024-03-21T08:42:29.686362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666959+0000","last_change":"2024-03-21T08:42:28.666959+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666959+0000","last_undegraded":"2024-03-21T08:42:28.666959+0000","last_fullsized":"2024-03-21T08:42:28.666959+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663336+0000","last_change":"2024-03-21T08:42:28.663336+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663336+0000","last_undegraded":"2024-03-21T08:42:28.663336+0000","last_fullsized":"2024-03-21T08:42:28.663336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695211+0000","last_change":"2024-03-21T08:42:29.689467+0000","last_active":"2024-03-21T08:42:29.695211+0000","last_peered":"2024-03-21T08:42:29.695211+0000","last_clean":"2024-03-21T08:42:29.695211+0000","last_became_active":"2024-03-21T08:42:29.688797+0000","last_became_peered":"2024-03-21T08:42:29.688797+0000","last_unstale":"2024-03-21T08:42:29.695211+0000","last_undegraded":"2024-03-21T08:42:29.695211+0000","last_fullsized":"2024-03-21T08:42:29.695211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692366+0000","last_change":"2024-03-21T08:42:29.688043+0000","last_active":"2024-03-21T08:42:29.692366+0000","last_peered":"2024-03-21T08:42:29.692366+0000","last_clean":"2024-03-21T08:42:29.692366+0000","last_became_active":"2024-03-21T08:42:29.686991+0000","last_became_peered":"2024-03-21T08:42:29.686991+0000","last_unstale":"2024-03-21T08:42:29.692366+0000","last_undegraded":"2024-03-21T08:42:29.692366+0000","last_fullsized":"2024-03-21T08:42:29.692366+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667123+0000","last_change":"2024-03-21T08:42:28.667123+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667123+0000","last_undegraded":"2024-03-21T08:42:28.667123+0000","last_fullsized":"2024-03-21T08:42:28.667123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670052+0000","last_change":"2024-03-21T08:42:29.670052+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670052+0000","last_undegraded":"2024-03-21T08:42:29.670052+0000","last_fullsized":"2024-03-21T08:42:29.670052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667552+0000","last_change":"2024-03-21T08:42:28.667552+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667552+0000","last_undegraded":"2024-03-21T08:42:28.667552+0000","last_fullsized":"2024-03-21T08:42:28.667552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673776+0000","last_change":"2024-03-21T08:42:29.673776+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673776+0000","last_undegraded":"2024-03-21T08:42:29.673776+0000","last_fullsized":"2024-03-21T08:42:29.673776+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663252+0000","last_change":"2024-03-21T08:42:28.663252+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663252+0000","last_undegraded":"2024-03-21T08:42:28.663252+0000","last_fullsized":"2024-03-21T08:42:28.663252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663279+0000","last_change":"2024-03-21T08:42:28.663279+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663279+0000","last_undegraded":"2024-03-21T08:42:28.663279+0000","last_fullsized":"2024-03-21T08:42:28.663279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667313+0000","last_change":"2024-03-21T08:42:28.667313+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667313+0000","last_undegraded":"2024-03-21T08:42:28.667313+0000","last_fullsized":"2024-03-21T08:42:28.667313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666685+0000","last_change":"2024-03-21T08:42:28.666685+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666685+0000","last_undegraded":"2024-03-21T08:42:28.666685+0000","last_fullsized":"2024-03-21T08:42:28.666685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692251+0000","last_change":"2024-03-21T08:42:29.687057+0000","last_active":"2024-03-21T08:42:29.692251+0000","last_peered":"2024-03-21T08:42:29.692251+0000","last_clean":"2024-03-21T08:42:29.692251+0000","last_became_active":"2024-03-21T08:42:29.686782+0000","last_became_peered":"2024-03-21T08:42:29.686782+0000","last_unstale":"2024-03-21T08:42:29.692251+0000","last_undegraded":"2024-03-21T08:42:29.692251+0000","last_fullsized":"2024-03-21T08:42:29.692251+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677102+0000","last_change":"2024-03-21T08:42:29.677102+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677102+0000","last_undegraded":"2024-03-21T08:42:29.677102+0000","last_fullsized":"2024-03-21T08:42:29.677102+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.689055+0000","last_change":"2024-03-21T08:42:29.685401+0000","last_active":"2024-03-21T08:42:29.689055+0000","last_peered":"2024-03-21T08:42:29.689055+0000","last_clean":"2024-03-21T08:42:29.689055+0000","last_became_active":"2024-03-21T08:42:29.683391+0000","last_became_peered":"2024-03-21T08:42:29.683391+0000","last_unstale":"2024-03-21T08:42:29.689055+0000","last_undegraded":"2024-03-21T08:42:29.689055+0000","last_fullsized":"2024-03-21T08:42:29.689055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:51:29.004767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695295+0000","last_change":"2024-03-21T08:42:29.689164+0000","last_active":"2024-03-21T08:42:29.695295+0000","last_peered":"2024-03-21T08:42:29.695295+0000","last_clean":"2024-03-21T08:42:29.695295+0000","last_became_active":"2024-03-21T08:42:29.688812+0000","last_became_peered":"2024-03-21T08:42:29.688812+0000","last_unstale":"2024-03-21T08:42:29.695295+0000","last_undegraded":"2024-03-21T08:42:29.695295+0000","last_fullsized":"2024-03-21T08:42:29.695295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663917+0000","last_change":"2024-03-21T08:42:28.663917+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663917+0000","last_undegraded":"2024-03-21T08:42:28.663917+0000","last_fullsized":"2024-03-21T08:42:28.663917+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667605+0000","last_change":"2024-03-21T08:42:28.667605+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667605+0000","last_undegraded":"2024-03-21T08:42:28.667605+0000","last_fullsized":"2024-03-21T08:42:28.667605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687260+0000","last_change":"2024-03-21T08:42:29.685315+0000","last_active":"2024-03-21T08:42:29.687260+0000","last_peered":"2024-03-21T08:42:29.687260+0000","last_clean":"2024-03-21T08:42:29.687260+0000","last_became_active":"2024-03-21T08:42:29.684654+0000","last_became_peered":"2024-03-21T08:42:29.684654+0000","last_unstale":"2024-03-21T08:42:29.687260+0000","last_undegraded":"2024-03-21T08:42:29.687260+0000","last_fullsized":"2024-03-21T08:42:29.687260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:01.696133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.669582+0000","last_change":"2024-03-21T08:42:29.669582+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.669582+0000","last_undegraded":"2024-03-21T08:42:29.669582+0000","last_fullsized":"2024-03-21T08:42:29.669582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683975+0000","last_change":"2024-03-21T08:42:29.682271+0000","last_active":"2024-03-21T08:42:29.683975+0000","last_peered":"2024-03-21T08:42:29.683975+0000","last_clean":"2024-03-21T08:42:29.683975+0000","last_became_active":"2024-03-21T08:42:29.681820+0000","last_became_peered":"2024-03-21T08:42:29.681820+0000","last_unstale":"2024-03-21T08:42:29.683975+0000","last_undegraded":"2024-03-21T08:42:29.683975+0000","last_fullsized":"2024-03-21T08:42:29.683975+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:42.529477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668445+0000","last_change":"2024-03-21T08:42:29.668445+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668445+0000","last_undegraded":"2024-03-21T08:42:29.668445+0000","last_fullsized":"2024-03-21T08:42:29.668445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683604+0000","last_change":"2024-03-21T08:42:29.681387+0000","last_active":"2024-03-21T08:42:29.683604+0000","last_peered":"2024-03-21T08:42:29.683604+0000","last_clean":"2024-03-21T08:42:29.683604+0000","last_became_active":"2024-03-21T08:42:29.681045+0000","last_became_peered":"2024-03-21T08:42:29.681045+0000","last_unstale":"2024-03-21T08:42:29.683604+0000","last_undegraded":"2024-03-21T08:42:29.683604+0000","last_fullsized":"2024-03-21T08:42:29.683604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:29:50.097289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688592+0000","last_change":"2024-03-21T08:42:29.686227+0000","last_active":"2024-03-21T08:42:29.688592+0000","last_peered":"2024-03-21T08:42:29.688592+0000","last_clean":"2024-03-21T08:42:29.688592+0000","last_became_active":"2024-03-21T08:42:29.685952+0000","last_became_peered":"2024-03-21T08:42:29.685952+0000","last_unstale":"2024-03-21T08:42:29.688592+0000","last_undegraded":"2024-03-21T08:42:29.688592+0000","last_fullsized":"2024-03-21T08:42:29.688592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:28.806090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695302+0000","last_change":"2024-03-21T08:42:29.689729+0000","last_active":"2024-03-21T08:42:29.695302+0000","last_peered":"2024-03-21T08:42:29.695302+0000","last_clean":"2024-03-21T08:42:29.695302+0000","last_became_active":"2024-03-21T08:42:29.689417+0000","last_became_peered":"2024-03-21T08:42:29.689417+0000","last_unstale":"2024-03-21T08:42:29.695302+0000","last_undegraded":"2024-03-21T08:42:29.695302+0000","last_fullsized":"2024-03-21T08:42:29.695302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683552+0000","last_change":"2024-03-21T08:42:29.681441+0000","last_active":"2024-03-21T08:42:29.683552+0000","last_peered":"2024-03-21T08:42:29.683552+0000","last_clean":"2024-03-21T08:42:29.683552+0000","last_became_active":"2024-03-21T08:42:29.680958+0000","last_became_peered":"2024-03-21T08:42:29.680958+0000","last_unstale":"2024-03-21T08:42:29.683552+0000","last_undegraded":"2024-03-21T08:42:29.683552+0000","last_fullsized":"2024-03-21T08:42:29.683552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:17.027344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671753+0000","last_change":"2024-03-21T08:42:29.671753+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671753+0000","last_undegraded":"2024-03-21T08:42:29.671753+0000","last_fullsized":"2024-03-21T08:42:29.671753+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663326+0000","last_change":"2024-03-21T08:42:28.663326+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663326+0000","last_undegraded":"2024-03-21T08:42:28.663326+0000","last_fullsized":"2024-03-21T08:42:28.663326+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673350+0000","last_change":"2024-03-21T08:42:29.673350+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673350+0000","last_undegraded":"2024-03-21T08:42:29.673350+0000","last_fullsized":"2024-03-21T08:42:29.673350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682669+0000","last_change":"2024-03-21T08:42:29.680363+0000","last_active":"2024-03-21T08:42:29.682669+0000","last_peered":"2024-03-21T08:42:29.682669+0000","last_clean":"2024-03-21T08:42:29.682669+0000","last_became_active":"2024-03-21T08:42:29.680096+0000","last_became_peered":"2024-03-21T08:42:29.680096+0000","last_unstale":"2024-03-21T08:42:29.682669+0000","last_undegraded":"2024-03-21T08:42:29.682669+0000","last_fullsized":"2024-03-21T08:42:29.682669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:10.546256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667369+0000","last_change":"2024-03-21T08:42:28.667369+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667369+0000","last_undegraded":"2024-03-21T08:42:28.667369+0000","last_fullsized":"2024-03-21T08:42:28.667369+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666938+0000","last_change":"2024-03-21T08:42:28.666938+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666938+0000","last_undegraded":"2024-03-21T08:42:28.666938+0000","last_fullsized":"2024-03-21T08:42:28.666938+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672543+0000","last_change":"2024-03-21T08:42:29.672543+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672543+0000","last_undegraded":"2024-03-21T08:42:29.672543+0000","last_fullsized":"2024-03-21T08:42:29.672543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687388+0000","last_change":"2024-03-21T08:42:29.681258+0000","last_active":"2024-03-21T08:42:29.687388+0000","last_peered":"2024-03-21T08:42:29.687388+0000","last_clean":"2024-03-21T08:42:29.687388+0000","last_became_active":"2024-03-21T08:42:29.680977+0000","last_became_peered":"2024-03-21T08:42:29.680977+0000","last_unstale":"2024-03-21T08:42:29.687388+0000","last_undegraded":"2024-03-21T08:42:29.687388+0000","last_fullsized":"2024-03-21T08:42:29.687388+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:36:11.691860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666969+0000","last_change":"2024-03-21T08:42:28.666969+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666969+0000","last_undegraded":"2024-03-21T08:42:28.666969+0000","last_fullsized":"2024-03-21T08:42:28.666969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.675868+0000","last_change":"2024-03-21T08:42:29.675868+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.675868+0000","last_undegraded":"2024-03-21T08:42:29.675868+0000","last_fullsized":"2024-03-21T08:42:29.675868+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666690+0000","last_change":"2024-03-21T08:42:28.666690+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666690+0000","last_undegraded":"2024-03-21T08:42:28.666690+0000","last_fullsized":"2024-03-21T08:42:28.666690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.697642+0000","last_change":"2024-03-21T08:42:29.692572+0000","last_active":"2024-03-21T08:42:29.697642+0000","last_peered":"2024-03-21T08:42:29.697642+0000","last_clean":"2024-03-21T08:42:29.697642+0000","last_became_active":"2024-03-21T08:42:29.692230+0000","last_became_peered":"2024-03-21T08:42:29.692230+0000","last_unstale":"2024-03-21T08:42:29.697642+0000","last_undegraded":"2024-03-21T08:42:29.697642+0000","last_fullsized":"2024-03-21T08:42:29.697642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:13.487644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682238+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:42:29.682238+0000","last_peered":"2024-03-21T08:42:29.682238+0000","last_clean":"2024-03-21T08:42:29.682238+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:42:29.682238+0000","last_undegraded":"2024-03-21T08:42:29.682238+0000","last_fullsized":"2024-03-21T08:42:29.682238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":768530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":417792,"data_stored":178162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":761863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:42:34.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:42:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 Processor -- start 2024-03-21T08:42:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 -- start start 2024-03-21T08:42:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f954415ba20 0x7f954406b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f95440cf500 0x7f954406b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95440cc1f0 con 0x7f954415f540 2024-03-21T08:42:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954415be00 con 0x7f954415ba20 2024-03-21T08:42:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9543200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95440cfa00 con 0x7f95440cf500 2024-03-21T08:42:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f954415ba20 0x7f954406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f954415ba20 0x7f954406b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43158/0 (socket says 172.21.15.28:43158) 2024-03-21T08:42:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 -- 172.21.15.28:0/2645675261 learned_addr learned my addr 172.21.15.28:0/2645675261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:34.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541400640 1 --2- 172.21.15.28:0/2645675261 >> v2:172.21.15.151:3300/0 conn(0x7f95440cf500 0x7f954406b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 -- 172.21.15.28:0/2645675261 >> v2:172.21.15.151:3300/0 conn(0x7f95440cf500 msgr2=0x7f954406b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 --2- 172.21.15.28:0/2645675261 >> v2:172.21.15.151:3300/0 conn(0x7f95440cf500 0x7f954406b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 -- 172.21.15.28:0/2645675261 >> v1:172.21.15.151:6789/0 conn(0x7f954415f540 legacy=0x7f95440ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.991+0000 7f9541e00640 1 -- 172.21.15.28:0/2645675261 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440cfce0 con 0x7f954415ba20 2024-03-21T08:42:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9541e00640 1 --2- 172.21.15.28:0/2645675261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415ba20 0x7f954406b170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f954406bfc0 tx=0x7f953401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd63ddc3595cfb10 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9540a00640 1 -- 172.21.15.28:0/2645675261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340aa650 con 0x7f954415ba20 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9540a00640 1 -- 172.21.15.28:0/2645675261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95340af070 con 0x7f954415ba20 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9540a00640 1 -- 172.21.15.28:0/2645675261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340aaa10 con 0x7f954415ba20 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- 172.21.15.28:0/2645675261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415ba20 msgr2=0x7f954406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 --2- 172.21.15.28:0/2645675261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415ba20 0x7f954406b170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f954406bfc0 tx=0x7f953401c4f0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- 172.21.15.28:0/2645675261 shutdown_connections 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 --2- 172.21.15.28:0/2645675261 >> v2:172.21.15.151:3300/0 conn(0x7f95440cf500 0x7f954406b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 --2- 172.21.15.28:0/2645675261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415ba20 0x7f954406b170 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- 172.21.15.28:0/2645675261 >> 172.21.15.28:0/2645675261 conn(0x7f95440b2280 msgr2=0x7f95440b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- 172.21.15.28:0/2645675261 shutdown_connections 2024-03-21T08:42:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- 172.21.15.28:0/2645675261 wait complete. 2024-03-21T08:42:34.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 Processor -- start 2024-03-21T08:42:34.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.995+0000 7f9543200640 1 -- start start 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95440cf500 0x7f954414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 0x7f9544152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9544168830 con 0x7f954415f540 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95441686b0 con 0x7f954415ba20 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95441689b0 con 0x7f95440cf500 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43172/0 (socket says 172.21.15.28:43172) 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 -- 172.21.15.28:0/931897197 learned_addr learned my addr 172.21.15.28:0/931897197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9541400640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 0x7f9544152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9541e00640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95440cf500 0x7f954414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 -- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95440cf500 msgr2=0x7f954414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95440cf500 0x7f954414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 -- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 msgr2=0x7f9544152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 0x7f9544152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 -- 172.21.15.28:0/931897197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954414b0f0 con 0x7f954415f540 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9541400640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 0x7f9544152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9542800640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f953c00c7a0 tx=0x7f953c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c003b80 con 0x7f954415f540 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c003d20 con 0x7f954415f540 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95441657e0 con 0x7f954415f540 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c00aa30 con 0x7f954415f540 2024-03-21T08:42:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9544165da0 con 0x7f954415f540 2024-03-21T08:42:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:34.999+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954406d210 con 0x7f954415f540 2024-03-21T08:42:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.003+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f953c00abd0 con 0x7f954415f540 2024-03-21T08:42:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.003+0000 7f953a000640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 0x7f9524069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.003+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f953c085f10 con 0x7f954415f540 2024-03-21T08:42:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.003+0000 7f9541e00640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 0x7f9524069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.003+0000 7f9541e00640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 0x7f9524069fe0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95340ae000 tx=0x7f95340a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.007+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f95440cc1f0 con 0x7f954415f540 2024-03-21T08:42:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.167+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f954406ece0 con 0x7f9524067b20 2024-03-21T08:42:35.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.175+0000 7f953a000640 1 -- 172.21.15.28:0/931897197 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+243214 (secure 0 0 0) 0x7f95340af040 con 0x7f9524067b20 2024-03-21T08:42:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 msgr2=0x7f9524069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 0x7f9524069fe0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95340ae000 tx=0x7f95340a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 msgr2=0x7f9544152890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f953c00c7a0 tx=0x7f953c00cc60 comp rx=0 tx=0).stop 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 shutdown_connections 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f954415f540 0x7f9544152890 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f954415ba20 0x7f9544152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9524067b20 0x7f9524069fe0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 --2- 172.21.15.28:0/931897197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95440cf500 0x7f954414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 >> 172.21.15.28:0/931897197 conn(0x7f95440b2280 msgr2=0x7f95440b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 shutdown_connections 2024-03-21T08:42:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.179+0000 7f9543200640 1 -- 172.21.15.28:0/931897197 wait complete. 2024-03-21T08:42:35.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-03-21T08:42:33.736599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232984,"kb_used_data":4316,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741736,"statfs":{"total":773094113280,"available":772855537664,"internally_reserved":0,"allocated":4419584,"data_stored":2547403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.076102"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671004+0000","last_change":"2024-03-21T08:42:29.671004+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671004+0000","last_undegraded":"2024-03-21T08:42:29.671004+0000","last_fullsized":"2024-03-21T08:42:29.671004+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670994+0000","last_change":"2024-03-21T08:42:29.670994+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670994+0000","last_undegraded":"2024-03-21T08:42:29.670994+0000","last_fullsized":"2024-03-21T08:42:29.670994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671282+0000","last_change":"2024-03-21T08:42:29.671282+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671282+0000","last_undegraded":"2024-03-21T08:42:29.671282+0000","last_fullsized":"2024-03-21T08:42:29.671282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670924+0000","last_change":"2024-03-21T08:42:29.670924+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670924+0000","last_undegraded":"2024-03-21T08:42:29.670924+0000","last_fullsized":"2024-03-21T08:42:29.670924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672336+0000","last_change":"2024-03-21T08:42:29.672336+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672336+0000","last_undegraded":"2024-03-21T08:42:29.672336+0000","last_fullsized":"2024-03-21T08:42:29.672336+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673666+0000","last_change":"2024-03-21T08:42:29.673666+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673666+0000","last_undegraded":"2024-03-21T08:42:29.673666+0000","last_fullsized":"2024-03-21T08:42:29.673666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670674+0000","last_change":"2024-03-21T08:42:29.670674+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670674+0000","last_undegraded":"2024-03-21T08:42:29.670674+0000","last_fullsized":"2024-03-21T08:42:29.670674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673436+0000","last_change":"2024-03-21T08:42:29.673436+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673436+0000","last_undegraded":"2024-03-21T08:42:29.673436+0000","last_fullsized":"2024-03-21T08:42:29.673436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677791+0000","last_change":"2024-03-21T08:42:29.677791+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677791+0000","last_undegraded":"2024-03-21T08:42:29.677791+0000","last_fullsized":"2024-03-21T08:42:29.677791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670239+0000","last_change":"2024-03-21T08:42:29.670239+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670239+0000","last_undegraded":"2024-03-21T08:42:29.670239+0000","last_fullsized":"2024-03-21T08:42:29.670239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671865+0000","last_change":"2024-03-21T08:42:29.671865+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671865+0000","last_undegraded":"2024-03-21T08:42:29.671865+0000","last_fullsized":"2024-03-21T08:42:29.671865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672819+0000","last_change":"2024-03-21T08:42:29.672819+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672819+0000","last_undegraded":"2024-03-21T08:42:29.672819+0000","last_fullsized":"2024-03-21T08:42:29.672819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671261+0000","last_change":"2024-03-21T08:42:29.671261+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671261+0000","last_undegraded":"2024-03-21T08:42:29.671261+0000","last_fullsized":"2024-03-21T08:42:29.671261+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672622+0000","last_change":"2024-03-21T08:42:29.672622+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672622+0000","last_undegraded":"2024-03-21T08:42:29.672622+0000","last_fullsized":"2024-03-21T08:42:29.672622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671020+0000","last_change":"2024-03-21T08:42:29.671020+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671020+0000","last_undegraded":"2024-03-21T08:42:29.671020+0000","last_fullsized":"2024-03-21T08:42:29.671020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668449+0000","last_change":"2024-03-21T08:42:29.668449+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668449+0000","last_undegraded":"2024-03-21T08:42:29.668449+0000","last_fullsized":"2024-03-21T08:42:29.668449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671972+0000","last_change":"2024-03-21T08:42:29.671972+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671972+0000","last_undegraded":"2024-03-21T08:42:29.671972+0000","last_fullsized":"2024-03-21T08:42:29.671972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668747+0000","last_change":"2024-03-21T08:42:29.668747+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668747+0000","last_undegraded":"2024-03-21T08:42:29.668747+0000","last_fullsized":"2024-03-21T08:42:29.668747+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688377+0000","last_change":"2024-03-21T08:42:29.683944+0000","last_active":"2024-03-21T08:42:29.688377+0000","last_peered":"2024-03-21T08:42:29.688377+0000","last_clean":"2024-03-21T08:42:29.688377+0000","last_became_active":"2024-03-21T08:42:29.683089+0000","last_became_peered":"2024-03-21T08:42:29.683089+0000","last_unstale":"2024-03-21T08:42:29.688377+0000","last_undegraded":"2024-03-21T08:42:29.688377+0000","last_fullsized":"2024-03-21T08:42:29.688377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:21:01.343294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672583+0000","last_change":"2024-03-21T08:42:29.672583+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672583+0000","last_undegraded":"2024-03-21T08:42:29.672583+0000","last_fullsized":"2024-03-21T08:42:29.672583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667838+0000","last_change":"2024-03-21T08:42:28.667838+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667838+0000","last_undegraded":"2024-03-21T08:42:28.667838+0000","last_fullsized":"2024-03-21T08:42:28.667838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.686362+0000","last_change":"2024-03-21T08:42:29.683368+0000","last_active":"2024-03-21T08:42:29.686362+0000","last_peered":"2024-03-21T08:42:29.686362+0000","last_clean":"2024-03-21T08:42:29.686362+0000","last_became_active":"2024-03-21T08:42:29.683095+0000","last_became_peered":"2024-03-21T08:42:29.683095+0000","last_unstale":"2024-03-21T08:42:29.686362+0000","last_undegraded":"2024-03-21T08:42:29.686362+0000","last_fullsized":"2024-03-21T08:42:29.686362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666959+0000","last_change":"2024-03-21T08:42:28.666959+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666959+0000","last_undegraded":"2024-03-21T08:42:28.666959+0000","last_fullsized":"2024-03-21T08:42:28.666959+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663336+0000","last_change":"2024-03-21T08:42:28.663336+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663336+0000","last_undegraded":"2024-03-21T08:42:28.663336+0000","last_fullsized":"2024-03-21T08:42:28.663336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695211+0000","last_change":"2024-03-21T08:42:29.689467+0000","last_active":"2024-03-21T08:42:29.695211+0000","last_peered":"2024-03-21T08:42:29.695211+0000","last_clean":"2024-03-21T08:42:29.695211+0000","last_became_active":"2024-03-21T08:42:29.688797+0000","last_became_peered":"2024-03-21T08:42:29.688797+0000","last_unstale":"2024-03-21T08:42:29.695211+0000","last_undegraded":"2024-03-21T08:42:29.695211+0000","last_fullsized":"2024-03-21T08:42:29.695211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692366+0000","last_change":"2024-03-21T08:42:29.688043+0000","last_active":"2024-03-21T08:42:29.692366+0000","last_peered":"2024-03-21T08:42:29.692366+0000","last_clean":"2024-03-21T08:42:29.692366+0000","last_became_active":"2024-03-21T08:42:29.686991+0000","last_became_peered":"2024-03-21T08:42:29.686991+0000","last_unstale":"2024-03-21T08:42:29.692366+0000","last_undegraded":"2024-03-21T08:42:29.692366+0000","last_fullsized":"2024-03-21T08:42:29.692366+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667123+0000","last_change":"2024-03-21T08:42:28.667123+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667123+0000","last_undegraded":"2024-03-21T08:42:28.667123+0000","last_fullsized":"2024-03-21T08:42:28.667123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670052+0000","last_change":"2024-03-21T08:42:29.670052+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670052+0000","last_undegraded":"2024-03-21T08:42:29.670052+0000","last_fullsized":"2024-03-21T08:42:29.670052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667552+0000","last_change":"2024-03-21T08:42:28.667552+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667552+0000","last_undegraded":"2024-03-21T08:42:28.667552+0000","last_fullsized":"2024-03-21T08:42:28.667552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673776+0000","last_change":"2024-03-21T08:42:29.673776+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673776+0000","last_undegraded":"2024-03-21T08:42:29.673776+0000","last_fullsized":"2024-03-21T08:42:29.673776+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663252+0000","last_change":"2024-03-21T08:42:28.663252+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663252+0000","last_undegraded":"2024-03-21T08:42:28.663252+0000","last_fullsized":"2024-03-21T08:42:28.663252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663279+0000","last_change":"2024-03-21T08:42:28.663279+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663279+0000","last_undegraded":"2024-03-21T08:42:28.663279+0000","last_fullsized":"2024-03-21T08:42:28.663279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667313+0000","last_change":"2024-03-21T08:42:28.667313+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667313+0000","last_undegraded":"2024-03-21T08:42:28.667313+0000","last_fullsized":"2024-03-21T08:42:28.667313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666685+0000","last_change":"2024-03-21T08:42:28.666685+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666685+0000","last_undegraded":"2024-03-21T08:42:28.666685+0000","last_fullsized":"2024-03-21T08:42:28.666685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692251+0000","last_change":"2024-03-21T08:42:29.687057+0000","last_active":"2024-03-21T08:42:29.692251+0000","last_peered":"2024-03-21T08:42:29.692251+0000","last_clean":"2024-03-21T08:42:29.692251+0000","last_became_active":"2024-03-21T08:42:29.686782+0000","last_became_peered":"2024-03-21T08:42:29.686782+0000","last_unstale":"2024-03-21T08:42:29.692251+0000","last_undegraded":"2024-03-21T08:42:29.692251+0000","last_fullsized":"2024-03-21T08:42:29.692251+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677102+0000","last_change":"2024-03-21T08:42:29.677102+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677102+0000","last_undegraded":"2024-03-21T08:42:29.677102+0000","last_fullsized":"2024-03-21T08:42:29.677102+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.689055+0000","last_change":"2024-03-21T08:42:29.685401+0000","last_active":"2024-03-21T08:42:29.689055+0000","last_peered":"2024-03-21T08:42:29.689055+0000","last_clean":"2024-03-21T08:42:29.689055+0000","last_became_active":"2024-03-21T08:42:29.683391+0000","last_became_peered":"2024-03-21T08:42:29.683391+0000","last_unstale":"2024-03-21T08:42:29.689055+0000","last_undegraded":"2024-03-21T08:42:29.689055+0000","last_fullsized":"2024-03-21T08:42:29.689055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:51:29.004767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695295+0000","last_change":"2024-03-21T08:42:29.689164+0000","last_active":"2024-03-21T08:42:29.695295+0000","last_peered":"2024-03-21T08:42:29.695295+0000","last_clean":"2024-03-21T08:42:29.695295+0000","last_became_active":"2024-03-21T08:42:29.688812+0000","last_became_peered":"2024-03-21T08:42:29.688812+0000","last_unstale":"2024-03-21T08:42:29.695295+0000","last_undegraded":"2024-03-21T08:42:29.695295+0000","last_fullsized":"2024-03-21T08:42:29.695295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663917+0000","last_change":"2024-03-21T08:42:28.663917+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663917+0000","last_undegraded":"2024-03-21T08:42:28.663917+0000","last_fullsized":"2024-03-21T08:42:28.663917+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667605+0000","last_change":"2024-03-21T08:42:28.667605+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667605+0000","last_undegraded":"2024-03-21T08:42:28.667605+0000","last_fullsized":"2024-03-21T08:42:28.667605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687260+0000","last_change":"2024-03-21T08:42:29.685315+0000","last_active":"2024-03-21T08:42:29.687260+0000","last_peered":"2024-03-21T08:42:29.687260+0000","last_clean":"2024-03-21T08:42:29.687260+0000","last_became_active":"2024-03-21T08:42:29.684654+0000","last_became_peered":"2024-03-21T08:42:29.684654+0000","last_unstale":"2024-03-21T08:42:29.687260+0000","last_undegraded":"2024-03-21T08:42:29.687260+0000","last_fullsized":"2024-03-21T08:42:29.687260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:01.696133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.669582+0000","last_change":"2024-03-21T08:42:29.669582+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.669582+0000","last_undegraded":"2024-03-21T08:42:29.669582+0000","last_fullsized":"2024-03-21T08:42:29.669582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683975+0000","last_change":"2024-03-21T08:42:29.682271+0000","last_active":"2024-03-21T08:42:29.683975+0000","last_peered":"2024-03-21T08:42:29.683975+0000","last_clean":"2024-03-21T08:42:29.683975+0000","last_became_active":"2024-03-21T08:42:29.681820+0000","last_became_peered":"2024-03-21T08:42:29.681820+0000","last_unstale":"2024-03-21T08:42:29.683975+0000","last_undegraded":"2024-03-21T08:42:29.683975+0000","last_fullsized":"2024-03-21T08:42:29.683975+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:42.529477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668445+0000","last_change":"2024-03-21T08:42:29.668445+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668445+0000","last_undegraded":"2024-03-21T08:42:29.668445+0000","last_fullsized":"2024-03-21T08:42:29.668445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683604+0000","last_change":"2024-03-21T08:42:29.681387+0000","last_active":"2024-03-21T08:42:29.683604+0000","last_peered":"2024-03-21T08:42:29.683604+0000","last_clean":"2024-03-21T08:42:29.683604+0000","last_became_active":"2024-03-21T08:42:29.681045+0000","last_became_peered":"2024-03-21T08:42:29.681045+0000","last_unstale":"2024-03-21T08:42:29.683604+0000","last_undegraded":"2024-03-21T08:42:29.683604+0000","last_fullsized":"2024-03-21T08:42:29.683604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:29:50.097289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688592+0000","last_change":"2024-03-21T08:42:29.686227+0000","last_active":"2024-03-21T08:42:29.688592+0000","last_peered":"2024-03-21T08:42:29.688592+0000","last_clean":"2024-03-21T08:42:29.688592+0000","last_became_active":"2024-03-21T08:42:29.685952+0000","last_became_peered":"2024-03-21T08:42:29.685952+0000","last_unstale":"2024-03-21T08:42:29.688592+0000","last_undegraded":"2024-03-21T08:42:29.688592+0000","last_fullsized":"2024-03-21T08:42:29.688592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:28.806090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695302+0000","last_change":"2024-03-21T08:42:29.689729+0000","last_active":"2024-03-21T08:42:29.695302+0000","last_peered":"2024-03-21T08:42:29.695302+0000","last_clean":"2024-03-21T08:42:29.695302+0000","last_became_active":"2024-03-21T08:42:29.689417+0000","last_became_peered":"2024-03-21T08:42:29.689417+0000","last_unstale":"2024-03-21T08:42:29.695302+0000","last_undegraded":"2024-03-21T08:42:29.695302+0000","last_fullsized":"2024-03-21T08:42:29.695302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683552+0000","last_change":"2024-03-21T08:42:29.681441+0000","last_active":"2024-03-21T08:42:29.683552+0000","last_peered":"2024-03-21T08:42:29.683552+0000","last_clean":"2024-03-21T08:42:29.683552+0000","last_became_active":"2024-03-21T08:42:29.680958+0000","last_became_peered":"2024-03-21T08:42:29.680958+0000","last_unstale":"2024-03-21T08:42:29.683552+0000","last_undegraded":"2024-03-21T08:42:29.683552+0000","last_fullsized":"2024-03-21T08:42:29.683552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:17.027344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671753+0000","last_change":"2024-03-21T08:42:29.671753+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671753+0000","last_undegraded":"2024-03-21T08:42:29.671753+0000","last_fullsized":"2024-03-21T08:42:29.671753+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663326+0000","last_change":"2024-03-21T08:42:28.663326+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663326+0000","last_undegraded":"2024-03-21T08:42:28.663326+0000","last_fullsized":"2024-03-21T08:42:28.663326+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673350+0000","last_change":"2024-03-21T08:42:29.673350+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673350+0000","last_undegraded":"2024-03-21T08:42:29.673350+0000","last_fullsized":"2024-03-21T08:42:29.673350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682669+0000","last_change":"2024-03-21T08:42:29.680363+0000","last_active":"2024-03-21T08:42:29.682669+0000","last_peered":"2024-03-21T08:42:29.682669+0000","last_clean":"2024-03-21T08:42:29.682669+0000","last_became_active":"2024-03-21T08:42:29.680096+0000","last_became_peered":"2024-03-21T08:42:29.680096+0000","last_unstale":"2024-03-21T08:42:29.682669+0000","last_undegraded":"2024-03-21T08:42:29.682669+0000","last_fullsized":"2024-03-21T08:42:29.682669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:10.546256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667369+0000","last_change":"2024-03-21T08:42:28.667369+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667369+0000","last_undegraded":"2024-03-21T08:42:28.667369+0000","last_fullsized":"2024-03-21T08:42:28.667369+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666938+0000","last_change":"2024-03-21T08:42:28.666938+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666938+0000","last_undegraded":"2024-03-21T08:42:28.666938+0000","last_fullsized":"2024-03-21T08:42:28.666938+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672543+0000","last_change":"2024-03-21T08:42:29.672543+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672543+0000","last_undegraded":"2024-03-21T08:42:29.672543+0000","last_fullsized":"2024-03-21T08:42:29.672543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687388+0000","last_change":"2024-03-21T08:42:29.681258+0000","last_active":"2024-03-21T08:42:29.687388+0000","last_peered":"2024-03-21T08:42:29.687388+0000","last_clean":"2024-03-21T08:42:29.687388+0000","last_became_active":"2024-03-21T08:42:29.680977+0000","last_became_peered":"2024-03-21T08:42:29.680977+0000","last_unstale":"2024-03-21T08:42:29.687388+0000","last_undegraded":"2024-03-21T08:42:29.687388+0000","last_fullsized":"2024-03-21T08:42:29.687388+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:36:11.691860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666969+0000","last_change":"2024-03-21T08:42:28.666969+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666969+0000","last_undegraded":"2024-03-21T08:42:28.666969+0000","last_fullsized":"2024-03-21T08:42:28.666969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.675868+0000","last_change":"2024-03-21T08:42:29.675868+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.675868+0000","last_undegraded":"2024-03-21T08:42:29.675868+0000","last_fullsized":"2024-03-21T08:42:29.675868+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666690+0000","last_change":"2024-03-21T08:42:28.666690+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666690+0000","last_undegraded":"2024-03-21T08:42:28.666690+0000","last_fullsized":"2024-03-21T08:42:28.666690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.697642+0000","last_change":"2024-03-21T08:42:29.692572+0000","last_active":"2024-03-21T08:42:29.697642+0000","last_peered":"2024-03-21T08:42:29.697642+0000","last_clean":"2024-03-21T08:42:29.697642+0000","last_became_active":"2024-03-21T08:42:29.692230+0000","last_became_peered":"2024-03-21T08:42:29.692230+0000","last_unstale":"2024-03-21T08:42:29.697642+0000","last_undegraded":"2024-03-21T08:42:29.697642+0000","last_fullsized":"2024-03-21T08:42:29.697642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:13.487644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682238+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:42:29.682238+0000","last_peered":"2024-03-21T08:42:29.682238+0000","last_clean":"2024-03-21T08:42:29.682238+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:42:29.682238+0000","last_undegraded":"2024-03-21T08:42:29.682238+0000","last_fullsized":"2024-03-21T08:42:29.682238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":768530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":417792,"data_stored":178162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":761863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:42:35.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:42:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.383+0000 7f84a4600640 1 Processor -- start 2024-03-21T08:42:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.383+0000 7f84a4600640 1 -- start start 2024-03-21T08:42:35.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a4600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f849c15e0c0 0x7f849c06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a4600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c0cc1d0 con 0x7f849c09a010 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a4600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c06d450 con 0x7f849c099b80 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a4600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849c0d1730 con 0x7f849c15e0c0 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f849c15e0c0 0x7f849c06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a2600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f849c09a010 0x7f849c0cd380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54510/0 (socket says 172.21.15.28:54510) 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f849c15e0c0 0x7f849c06cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43182/0 (socket says 172.21.15.28:43182) 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a2600640 1 -- 172.21.15.28:0/1266982582 learned_addr learned my addr 172.21.15.28:0/1266982582 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a1c00640 1 -- 172.21.15.28:0/1266982582 >> v1:172.21.15.151:6789/0 conn(0x7f849c099b80 legacy=0x7f849c0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a1c00640 1 -- 172.21.15.28:0/1266982582 >> v1:172.21.15.28:6789/0 conn(0x7f849c09a010 legacy=0x7f849c0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f849be00640 1 -- 172.21.15.28:0/1266982582 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55750536 0 0) 0x7f849c0cc1d0 con 0x7f849c09a010 2024-03-21T08:42:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.387+0000 7f84a1c00640 1 -- 172.21.15.28:0/1266982582 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0d1450 con 0x7f849c15e0c0 2024-03-21T08:42:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a1c00640 1 --2- 172.21.15.28:0/1266982582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c06cf10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f84900097c0 tx=0x7f849001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceed69ed4dd88be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f849be00640 1 -- 172.21.15.28:0/1266982582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84900b2070 con 0x7f849c15e0c0 2024-03-21T08:42:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f849be00640 1 -- 172.21.15.28:0/1266982582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84900ad680 con 0x7f849c15e0c0 2024-03-21T08:42:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f849be00640 1 -- 172.21.15.28:0/1266982582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84900ada50 con 0x7f849c15e0c0 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 -- 172.21.15.28:0/1266982582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 msgr2=0x7f849c06cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 --2- 172.21.15.28:0/1266982582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c06cf10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f84900097c0 tx=0x7f849001c470 comp rx=0 tx=0).stop 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 -- 172.21.15.28:0/1266982582 shutdown_connections 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 --2- 172.21.15.28:0/1266982582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c06cf10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 -- 172.21.15.28:0/1266982582 >> 172.21.15.28:0/1266982582 conn(0x7f849c0b34d0 msgr2=0x7f849c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 -- 172.21.15.28:0/1266982582 shutdown_connections 2024-03-21T08:42:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.391+0000 7f84a4600640 1 -- 172.21.15.28:0/1266982582 wait complete. 2024-03-21T08:42:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 Processor -- start 2024-03-21T08:42:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- start start 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f849c099b80 0x7f849c15e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f849c09a010 0x7f849c15f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0d10d0 con 0x7f849c15e0c0 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c15e4a0 con 0x7f849c09a010 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c16c4b0 con 0x7f849c099b80 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43188/0 (socket says 172.21.15.28:43188) 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 -- 172.21.15.28:0/1237797682 learned_addr learned my addr 172.21.15.28:0/1237797682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a1200640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f849c09a010 0x7f849c15f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a1c00640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f849c099b80 0x7f849c15e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 -- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f849c099b80 msgr2=0x7f849c15e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f849c099b80 0x7f849c15e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 -- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f849c09a010 msgr2=0x7f849c15f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f849c09a010 0x7f849c15f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 -- 172.21.15.28:0/1237797682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c161380 con 0x7f849c15e0c0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a2600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f849400c7d0 tx=0x7f849400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494018070 con 0x7f849c15e0c0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849400a640 con 0x7f849c15e0c0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849400aa70 con 0x7f849c15e0c0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c161660 con 0x7f849c15e0c0 2024-03-21T08:42:35.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.395+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c15afe0 con 0x7f849c15e0c0 2024-03-21T08:42:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c04f410 con 0x7f849c15e0c0 2024-03-21T08:42:35.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f849400ac10 con 0x7f849c15e0c0 2024-03-21T08:42:35.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f8499600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 0x7f847c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8494085660 con 0x7f849c15e0c0 2024-03-21T08:42:35.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f84a1c00640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 0x7f847c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.399+0000 7f84a1c00640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 0x7f847c069fe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84900af000 tx=0x7f84900a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.403+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8494055880 con 0x7f849c15e0c0 2024-03-21T08:42:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.595+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f849c0cce50 con 0x7f847c067b20 2024-03-21T08:42:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f8499600640 1 -- 172.21.15.28:0/1237797682 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+243214 (secure 0 0 0) 0x7f849c0cce50 con 0x7f847c067b20 2024-03-21T08:42:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:42:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 msgr2=0x7f847c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 0x7f847c069fe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84900af000 tx=0x7f84900a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 msgr2=0x7f849c15f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f849400c7d0 tx=0x7f849400cc90 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 shutdown_connections 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c15e0c0 0x7f849c15f570 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f849c09a010 0x7f849c15f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f847c067b20 0x7f847c069fe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 --2- 172.21.15.28:0/1237797682 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f849c099b80 0x7f849c15e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 >> 172.21.15.28:0/1237797682 conn(0x7f849c0b34d0 msgr2=0x7f849c0cd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 shutdown_connections 2024-03-21T08:42:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.603+0000 7f84a4600640 1 -- 172.21.15.28:0/1237797682 wait complete. 2024-03-21T08:42:35.628 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-03-21T08:42:33.736599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232984,"kb_used_data":4316,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741736,"statfs":{"total":773094113280,"available":772855537664,"internally_reserved":0,"allocated":4419584,"data_stored":2547403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.076102"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671004+0000","last_change":"2024-03-21T08:42:29.671004+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671004+0000","last_undegraded":"2024-03-21T08:42:29.671004+0000","last_fullsized":"2024-03-21T08:42:29.671004+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670994+0000","last_change":"2024-03-21T08:42:29.670994+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670994+0000","last_undegraded":"2024-03-21T08:42:29.670994+0000","last_fullsized":"2024-03-21T08:42:29.670994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671282+0000","last_change":"2024-03-21T08:42:29.671282+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671282+0000","last_undegraded":"2024-03-21T08:42:29.671282+0000","last_fullsized":"2024-03-21T08:42:29.671282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670924+0000","last_change":"2024-03-21T08:42:29.670924+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670924+0000","last_undegraded":"2024-03-21T08:42:29.670924+0000","last_fullsized":"2024-03-21T08:42:29.670924+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672336+0000","last_change":"2024-03-21T08:42:29.672336+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672336+0000","last_undegraded":"2024-03-21T08:42:29.672336+0000","last_fullsized":"2024-03-21T08:42:29.672336+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673666+0000","last_change":"2024-03-21T08:42:29.673666+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673666+0000","last_undegraded":"2024-03-21T08:42:29.673666+0000","last_fullsized":"2024-03-21T08:42:29.673666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670674+0000","last_change":"2024-03-21T08:42:29.670674+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670674+0000","last_undegraded":"2024-03-21T08:42:29.670674+0000","last_fullsized":"2024-03-21T08:42:29.670674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673436+0000","last_change":"2024-03-21T08:42:29.673436+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673436+0000","last_undegraded":"2024-03-21T08:42:29.673436+0000","last_fullsized":"2024-03-21T08:42:29.673436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677791+0000","last_change":"2024-03-21T08:42:29.677791+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677791+0000","last_undegraded":"2024-03-21T08:42:29.677791+0000","last_fullsized":"2024-03-21T08:42:29.677791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670239+0000","last_change":"2024-03-21T08:42:29.670239+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670239+0000","last_undegraded":"2024-03-21T08:42:29.670239+0000","last_fullsized":"2024-03-21T08:42:29.670239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671865+0000","last_change":"2024-03-21T08:42:29.671865+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671865+0000","last_undegraded":"2024-03-21T08:42:29.671865+0000","last_fullsized":"2024-03-21T08:42:29.671865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672819+0000","last_change":"2024-03-21T08:42:29.672819+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672819+0000","last_undegraded":"2024-03-21T08:42:29.672819+0000","last_fullsized":"2024-03-21T08:42:29.672819+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671261+0000","last_change":"2024-03-21T08:42:29.671261+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671261+0000","last_undegraded":"2024-03-21T08:42:29.671261+0000","last_fullsized":"2024-03-21T08:42:29.671261+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672622+0000","last_change":"2024-03-21T08:42:29.672622+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672622+0000","last_undegraded":"2024-03-21T08:42:29.672622+0000","last_fullsized":"2024-03-21T08:42:29.672622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671020+0000","last_change":"2024-03-21T08:42:29.671020+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671020+0000","last_undegraded":"2024-03-21T08:42:29.671020+0000","last_fullsized":"2024-03-21T08:42:29.671020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668449+0000","last_change":"2024-03-21T08:42:29.668449+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668449+0000","last_undegraded":"2024-03-21T08:42:29.668449+0000","last_fullsized":"2024-03-21T08:42:29.668449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671972+0000","last_change":"2024-03-21T08:42:29.671972+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671972+0000","last_undegraded":"2024-03-21T08:42:29.671972+0000","last_fullsized":"2024-03-21T08:42:29.671972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668747+0000","last_change":"2024-03-21T08:42:29.668747+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668747+0000","last_undegraded":"2024-03-21T08:42:29.668747+0000","last_fullsized":"2024-03-21T08:42:29.668747+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688377+0000","last_change":"2024-03-21T08:42:29.683944+0000","last_active":"2024-03-21T08:42:29.688377+0000","last_peered":"2024-03-21T08:42:29.688377+0000","last_clean":"2024-03-21T08:42:29.688377+0000","last_became_active":"2024-03-21T08:42:29.683089+0000","last_became_peered":"2024-03-21T08:42:29.683089+0000","last_unstale":"2024-03-21T08:42:29.688377+0000","last_undegraded":"2024-03-21T08:42:29.688377+0000","last_fullsized":"2024-03-21T08:42:29.688377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:21:01.343294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672583+0000","last_change":"2024-03-21T08:42:29.672583+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672583+0000","last_undegraded":"2024-03-21T08:42:29.672583+0000","last_fullsized":"2024-03-21T08:42:29.672583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667838+0000","last_change":"2024-03-21T08:42:28.667838+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667838+0000","last_undegraded":"2024-03-21T08:42:28.667838+0000","last_fullsized":"2024-03-21T08:42:28.667838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.686362+0000","last_change":"2024-03-21T08:42:29.683368+0000","last_active":"2024-03-21T08:42:29.686362+0000","last_peered":"2024-03-21T08:42:29.686362+0000","last_clean":"2024-03-21T08:42:29.686362+0000","last_became_active":"2024-03-21T08:42:29.683095+0000","last_became_peered":"2024-03-21T08:42:29.683095+0000","last_unstale":"2024-03-21T08:42:29.686362+0000","last_undegraded":"2024-03-21T08:42:29.686362+0000","last_fullsized":"2024-03-21T08:42:29.686362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666959+0000","last_change":"2024-03-21T08:42:28.666959+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666959+0000","last_undegraded":"2024-03-21T08:42:28.666959+0000","last_fullsized":"2024-03-21T08:42:28.666959+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663336+0000","last_change":"2024-03-21T08:42:28.663336+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663336+0000","last_undegraded":"2024-03-21T08:42:28.663336+0000","last_fullsized":"2024-03-21T08:42:28.663336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695211+0000","last_change":"2024-03-21T08:42:29.689467+0000","last_active":"2024-03-21T08:42:29.695211+0000","last_peered":"2024-03-21T08:42:29.695211+0000","last_clean":"2024-03-21T08:42:29.695211+0000","last_became_active":"2024-03-21T08:42:29.688797+0000","last_became_peered":"2024-03-21T08:42:29.688797+0000","last_unstale":"2024-03-21T08:42:29.695211+0000","last_undegraded":"2024-03-21T08:42:29.695211+0000","last_fullsized":"2024-03-21T08:42:29.695211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692366+0000","last_change":"2024-03-21T08:42:29.688043+0000","last_active":"2024-03-21T08:42:29.692366+0000","last_peered":"2024-03-21T08:42:29.692366+0000","last_clean":"2024-03-21T08:42:29.692366+0000","last_became_active":"2024-03-21T08:42:29.686991+0000","last_became_peered":"2024-03-21T08:42:29.686991+0000","last_unstale":"2024-03-21T08:42:29.692366+0000","last_undegraded":"2024-03-21T08:42:29.692366+0000","last_fullsized":"2024-03-21T08:42:29.692366+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667123+0000","last_change":"2024-03-21T08:42:28.667123+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667123+0000","last_undegraded":"2024-03-21T08:42:28.667123+0000","last_fullsized":"2024-03-21T08:42:28.667123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.670052+0000","last_change":"2024-03-21T08:42:29.670052+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.670052+0000","last_undegraded":"2024-03-21T08:42:29.670052+0000","last_fullsized":"2024-03-21T08:42:29.670052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667552+0000","last_change":"2024-03-21T08:42:28.667552+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667552+0000","last_undegraded":"2024-03-21T08:42:28.667552+0000","last_fullsized":"2024-03-21T08:42:28.667552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673776+0000","last_change":"2024-03-21T08:42:29.673776+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673776+0000","last_undegraded":"2024-03-21T08:42:29.673776+0000","last_fullsized":"2024-03-21T08:42:29.673776+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663252+0000","last_change":"2024-03-21T08:42:28.663252+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663252+0000","last_undegraded":"2024-03-21T08:42:28.663252+0000","last_fullsized":"2024-03-21T08:42:28.663252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663279+0000","last_change":"2024-03-21T08:42:28.663279+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663279+0000","last_undegraded":"2024-03-21T08:42:28.663279+0000","last_fullsized":"2024-03-21T08:42:28.663279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667313+0000","last_change":"2024-03-21T08:42:28.667313+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667313+0000","last_undegraded":"2024-03-21T08:42:28.667313+0000","last_fullsized":"2024-03-21T08:42:28.667313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666685+0000","last_change":"2024-03-21T08:42:28.666685+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666685+0000","last_undegraded":"2024-03-21T08:42:28.666685+0000","last_fullsized":"2024-03-21T08:42:28.666685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.692251+0000","last_change":"2024-03-21T08:42:29.687057+0000","last_active":"2024-03-21T08:42:29.692251+0000","last_peered":"2024-03-21T08:42:29.692251+0000","last_clean":"2024-03-21T08:42:29.692251+0000","last_became_active":"2024-03-21T08:42:29.686782+0000","last_became_peered":"2024-03-21T08:42:29.686782+0000","last_unstale":"2024-03-21T08:42:29.692251+0000","last_undegraded":"2024-03-21T08:42:29.692251+0000","last_fullsized":"2024-03-21T08:42:29.692251+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.677102+0000","last_change":"2024-03-21T08:42:29.677102+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.677102+0000","last_undegraded":"2024-03-21T08:42:29.677102+0000","last_fullsized":"2024-03-21T08:42:29.677102+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.689055+0000","last_change":"2024-03-21T08:42:29.685401+0000","last_active":"2024-03-21T08:42:29.689055+0000","last_peered":"2024-03-21T08:42:29.689055+0000","last_clean":"2024-03-21T08:42:29.689055+0000","last_became_active":"2024-03-21T08:42:29.683391+0000","last_became_peered":"2024-03-21T08:42:29.683391+0000","last_unstale":"2024-03-21T08:42:29.689055+0000","last_undegraded":"2024-03-21T08:42:29.689055+0000","last_fullsized":"2024-03-21T08:42:29.689055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:51:29.004767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695295+0000","last_change":"2024-03-21T08:42:29.689164+0000","last_active":"2024-03-21T08:42:29.695295+0000","last_peered":"2024-03-21T08:42:29.695295+0000","last_clean":"2024-03-21T08:42:29.695295+0000","last_became_active":"2024-03-21T08:42:29.688812+0000","last_became_peered":"2024-03-21T08:42:29.688812+0000","last_unstale":"2024-03-21T08:42:29.695295+0000","last_undegraded":"2024-03-21T08:42:29.695295+0000","last_fullsized":"2024-03-21T08:42:29.695295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663917+0000","last_change":"2024-03-21T08:42:28.663917+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663917+0000","last_undegraded":"2024-03-21T08:42:28.663917+0000","last_fullsized":"2024-03-21T08:42:28.663917+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667605+0000","last_change":"2024-03-21T08:42:28.667605+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667605+0000","last_undegraded":"2024-03-21T08:42:28.667605+0000","last_fullsized":"2024-03-21T08:42:28.667605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687260+0000","last_change":"2024-03-21T08:42:29.685315+0000","last_active":"2024-03-21T08:42:29.687260+0000","last_peered":"2024-03-21T08:42:29.687260+0000","last_clean":"2024-03-21T08:42:29.687260+0000","last_became_active":"2024-03-21T08:42:29.684654+0000","last_became_peered":"2024-03-21T08:42:29.684654+0000","last_unstale":"2024-03-21T08:42:29.687260+0000","last_undegraded":"2024-03-21T08:42:29.687260+0000","last_fullsized":"2024-03-21T08:42:29.687260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:01.696133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.669582+0000","last_change":"2024-03-21T08:42:29.669582+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.669582+0000","last_undegraded":"2024-03-21T08:42:29.669582+0000","last_fullsized":"2024-03-21T08:42:29.669582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683975+0000","last_change":"2024-03-21T08:42:29.682271+0000","last_active":"2024-03-21T08:42:29.683975+0000","last_peered":"2024-03-21T08:42:29.683975+0000","last_clean":"2024-03-21T08:42:29.683975+0000","last_became_active":"2024-03-21T08:42:29.681820+0000","last_became_peered":"2024-03-21T08:42:29.681820+0000","last_unstale":"2024-03-21T08:42:29.683975+0000","last_undegraded":"2024-03-21T08:42:29.683975+0000","last_fullsized":"2024-03-21T08:42:29.683975+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:42.529477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.668445+0000","last_change":"2024-03-21T08:42:29.668445+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.668445+0000","last_undegraded":"2024-03-21T08:42:29.668445+0000","last_fullsized":"2024-03-21T08:42:29.668445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683604+0000","last_change":"2024-03-21T08:42:29.681387+0000","last_active":"2024-03-21T08:42:29.683604+0000","last_peered":"2024-03-21T08:42:29.683604+0000","last_clean":"2024-03-21T08:42:29.683604+0000","last_became_active":"2024-03-21T08:42:29.681045+0000","last_became_peered":"2024-03-21T08:42:29.681045+0000","last_unstale":"2024-03-21T08:42:29.683604+0000","last_undegraded":"2024-03-21T08:42:29.683604+0000","last_fullsized":"2024-03-21T08:42:29.683604+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:29:50.097289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.688592+0000","last_change":"2024-03-21T08:42:29.686227+0000","last_active":"2024-03-21T08:42:29.688592+0000","last_peered":"2024-03-21T08:42:29.688592+0000","last_clean":"2024-03-21T08:42:29.688592+0000","last_became_active":"2024-03-21T08:42:29.685952+0000","last_became_peered":"2024-03-21T08:42:29.685952+0000","last_unstale":"2024-03-21T08:42:29.688592+0000","last_undegraded":"2024-03-21T08:42:29.688592+0000","last_fullsized":"2024-03-21T08:42:29.688592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:55:28.806090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.695302+0000","last_change":"2024-03-21T08:42:29.689729+0000","last_active":"2024-03-21T08:42:29.695302+0000","last_peered":"2024-03-21T08:42:29.695302+0000","last_clean":"2024-03-21T08:42:29.695302+0000","last_became_active":"2024-03-21T08:42:29.689417+0000","last_became_peered":"2024-03-21T08:42:29.689417+0000","last_unstale":"2024-03-21T08:42:29.695302+0000","last_undegraded":"2024-03-21T08:42:29.695302+0000","last_fullsized":"2024-03-21T08:42:29.695302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:56.858486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.683552+0000","last_change":"2024-03-21T08:42:29.681441+0000","last_active":"2024-03-21T08:42:29.683552+0000","last_peered":"2024-03-21T08:42:29.683552+0000","last_clean":"2024-03-21T08:42:29.683552+0000","last_became_active":"2024-03-21T08:42:29.680958+0000","last_became_peered":"2024-03-21T08:42:29.680958+0000","last_unstale":"2024-03-21T08:42:29.683552+0000","last_undegraded":"2024-03-21T08:42:29.683552+0000","last_fullsized":"2024-03-21T08:42:29.683552+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:17.027344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.671753+0000","last_change":"2024-03-21T08:42:29.671753+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.671753+0000","last_undegraded":"2024-03-21T08:42:29.671753+0000","last_fullsized":"2024-03-21T08:42:29.671753+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.663326+0000","last_change":"2024-03-21T08:42:28.663326+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.663326+0000","last_undegraded":"2024-03-21T08:42:28.663326+0000","last_fullsized":"2024-03-21T08:42:28.663326+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.673350+0000","last_change":"2024-03-21T08:42:29.673350+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.673350+0000","last_undegraded":"2024-03-21T08:42:29.673350+0000","last_fullsized":"2024-03-21T08:42:29.673350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682669+0000","last_change":"2024-03-21T08:42:29.680363+0000","last_active":"2024-03-21T08:42:29.682669+0000","last_peered":"2024-03-21T08:42:29.682669+0000","last_clean":"2024-03-21T08:42:29.682669+0000","last_became_active":"2024-03-21T08:42:29.680096+0000","last_became_peered":"2024-03-21T08:42:29.680096+0000","last_unstale":"2024-03-21T08:42:29.682669+0000","last_undegraded":"2024-03-21T08:42:29.682669+0000","last_fullsized":"2024-03-21T08:42:29.682669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:46:10.546256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.667369+0000","last_change":"2024-03-21T08:42:28.667369+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.667369+0000","last_undegraded":"2024-03-21T08:42:28.667369+0000","last_fullsized":"2024-03-21T08:42:28.667369+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666938+0000","last_change":"2024-03-21T08:42:28.666938+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666938+0000","last_undegraded":"2024-03-21T08:42:28.666938+0000","last_fullsized":"2024-03-21T08:42:28.666938+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.672543+0000","last_change":"2024-03-21T08:42:29.672543+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.672543+0000","last_undegraded":"2024-03-21T08:42:29.672543+0000","last_fullsized":"2024-03-21T08:42:29.672543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.687388+0000","last_change":"2024-03-21T08:42:29.681258+0000","last_active":"2024-03-21T08:42:29.687388+0000","last_peered":"2024-03-21T08:42:29.687388+0000","last_clean":"2024-03-21T08:42:29.687388+0000","last_became_active":"2024-03-21T08:42:29.680977+0000","last_became_peered":"2024-03-21T08:42:29.680977+0000","last_unstale":"2024-03-21T08:42:29.687388+0000","last_undegraded":"2024-03-21T08:42:29.687388+0000","last_fullsized":"2024-03-21T08:42:29.687388+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:36:11.691860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666969+0000","last_change":"2024-03-21T08:42:28.666969+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666969+0000","last_undegraded":"2024-03-21T08:42:28.666969+0000","last_fullsized":"2024-03-21T08:42:28.666969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-21T08:42:29.675868+0000","last_change":"2024-03-21T08:42:29.675868+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.675868+0000","last_undegraded":"2024-03-21T08:42:29.675868+0000","last_fullsized":"2024-03-21T08:42:29.675868+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-03-21T08:42:28.666690+0000","last_change":"2024-03-21T08:42:28.666690+0000","last_active":"2024-03-21T08:42:28.653731+0000","last_peered":"2024-03-21T08:42:28.653731+0000","last_clean":"2024-03-21T08:42:28.653731+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:28.666690+0000","last_undegraded":"2024-03-21T08:42:28.666690+0000","last_fullsized":"2024-03-21T08:42:28.666690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T08:42:29.660854+0000","last_change":"2024-03-21T08:42:29.660854+0000","last_active":"2024-03-21T08:42:29.660854+0000","last_peered":"2024-03-21T08:42:29.660854+0000","last_clean":"2024-03-21T08:42:29.660854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:42:29.660854+0000","last_undegraded":"2024-03-21T08:42:29.660854+0000","last_fullsized":"2024-03-21T08:42:29.660854+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.697642+0000","last_change":"2024-03-21T08:42:29.692572+0000","last_active":"2024-03-21T08:42:29.697642+0000","last_peered":"2024-03-21T08:42:29.697642+0000","last_clean":"2024-03-21T08:42:29.697642+0000","last_became_active":"2024-03-21T08:42:29.692230+0000","last_became_peered":"2024-03-21T08:42:29.692230+0000","last_unstale":"2024-03-21T08:42:29.697642+0000","last_undegraded":"2024-03-21T08:42:29.697642+0000","last_fullsized":"2024-03-21T08:42:29.697642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:13.487644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-21T08:42:29.682238+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:42:29.682238+0000","last_peered":"2024-03-21T08:42:29.682238+0000","last_clean":"2024-03-21T08:42:29.682238+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:42:29.682238+0000","last_undegraded":"2024-03-21T08:42:29.682238+0000","last_fullsized":"2024-03-21T08:42:29.682238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":768530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":171495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":417792,"data_stored":178162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":393216,"data_stored":165286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":761863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671004+0000', 'last_change': '2024-03-21T08:42:29.671004+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671004+0000', 'last_undegraded': '2024-03-21T08:42:29.671004+0000', 'last_fullsized': '2024-03-21T08:42:29.671004+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-03-21T08:42:35.632 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.670994+0000', 'last_change': '2024-03-21T08:42:29.670994+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.670994+0000', 'last_undegraded': '2024-03-21T08:42:29.670994+0000', 'last_fullsized': '2024-03-21T08:42:29.670994+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671282+0000', 'last_change': '2024-03-21T08:42:29.671282+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671282+0000', 'last_undegraded': '2024-03-21T08:42:29.671282+0000', 'last_fullsized': '2024-03-21T08:42:29.671282+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-03-21T08:42:35.633 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-03-21T08:42:35.634 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.670924+0000', 'last_change': '2024-03-21T08:42:29.670924+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.670924+0000', 'last_undegraded': '2024-03-21T08:42:29.670924+0000', 'last_fullsized': '2024-03-21T08:42:29.670924+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.672336+0000', 'last_change': '2024-03-21T08:42:29.672336+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.672336+0000', 'last_undegraded': '2024-03-21T08:42:29.672336+0000', 'last_fullsized': '2024-03-21T08:42:29.672336+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.673666+0000', 'last_change': '2024-03-21T08:42:29.673666+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.673666+0000', 'last_undegraded': '2024-03-21T08:42:29.673666+0000', 'last_fullsized': '2024-03-21T08:42:29.673666+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-03-21T08:42:35.635 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.670674+0000', 'last_change': '2024-03-21T08:42:29.670674+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.670674+0000', 'last_undegraded': '2024-03-21T08:42:29.670674+0000', 'last_fullsized': '2024-03-21T08:42:29.670674+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.673436+0000', 'last_change': '2024-03-21T08:42:29.673436+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.673436+0000', 'last_undegraded': '2024-03-21T08:42:29.673436+0000', 'last_fullsized': '2024-03-21T08:42:29.673436+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.677791+0000', 'last_change': '2024-03-21T08:42:29.677791+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.677791+0000', 'last_undegraded': '2024-03-21T08:42:29.677791+0000', 'last_fullsized': '2024-03-21T08:42:29.677791+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-03-21T08:42:35.636 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.670239+0000', 'last_change': '2024-03-21T08:42:29.670239+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.670239+0000', 'last_undegraded': '2024-03-21T08:42:29.670239+0000', 'last_fullsized': '2024-03-21T08:42:29.670239+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671865+0000', 'last_change': '2024-03-21T08:42:29.671865+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671865+0000', 'last_undegraded': '2024-03-21T08:42:29.671865+0000', 'last_fullsized': '2024-03-21T08:42:29.671865+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.672819+0000', 'last_change': '2024-03-21T08:42:29.672819+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.672819+0000', 'last_undegraded': '2024-03-21T08:42:29.672819+0000', 'last_fullsized': '2024-03-21T08:42:29.672819+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671261+0000', 'last_change': '2024-03-21T08:42:29.671261+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671261+0000', 'last_undegraded': '2024-03-21T08:42:29.671261+0000', 'last_fullsized': '2024-03-21T08:42:29.671261+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.672622+0000', 'last_change': '2024-03-21T08:42:29.672622+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.672622+0000', 'last_undegraded': '2024-03-21T08:42:29.672622+0000', 'last_fullsized': '2024-03-21T08:42:29.672622+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.637 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671020+0000', 'last_change': '2024-03-21T08:42:29.671020+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671020+0000', 'last_undegraded': '2024-03-21T08:42:29.671020+0000', 'last_fullsized': '2024-03-21T08:42:29.671020+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.668449+0000', 'last_change': '2024-03-21T08:42:29.668449+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.668449+0000', 'last_undegraded': '2024-03-21T08:42:29.668449+0000', 'last_fullsized': '2024-03-21T08:42:29.668449+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671972+0000', 'last_change': '2024-03-21T08:42:29.671972+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671972+0000', 'last_undegraded': '2024-03-21T08:42:29.671972+0000', 'last_fullsized': '2024-03-21T08:42:29.671972+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.668747+0000', 'last_change': '2024-03-21T08:42:29.668747+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.668747+0000', 'last_undegraded': '2024-03-21T08:42:29.668747+0000', 'last_fullsized': '2024-03-21T08:42:29.668747+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.672583+0000', 'last_change': '2024-03-21T08:42:29.672583+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.672583+0000', 'last_undegraded': '2024-03-21T08:42:29.672583+0000', 'last_fullsized': '2024-03-21T08:42:29.672583+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-03-21T08:42:35.638 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667838+0000', 'last_change': '2024-03-21T08:42:28.667838+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667838+0000', 'last_undegraded': '2024-03-21T08:42:28.667838+0000', 'last_fullsized': '2024-03-21T08:42:28.667838+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.666959+0000', 'last_change': '2024-03-21T08:42:28.666959+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.666959+0000', 'last_undegraded': '2024-03-21T08:42:28.666959+0000', 'last_fullsized': '2024-03-21T08:42:28.666959+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-03-21T08:42:35.639 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.663336+0000', 'last_change': '2024-03-21T08:42:28.663336+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.663336+0000', 'last_undegraded': '2024-03-21T08:42:28.663336+0000', 'last_fullsized': '2024-03-21T08:42:28.663336+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667123+0000', 'last_change': '2024-03-21T08:42:28.667123+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667123+0000', 'last_undegraded': '2024-03-21T08:42:28.667123+0000', 'last_fullsized': '2024-03-21T08:42:28.667123+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.670052+0000', 'last_change': '2024-03-21T08:42:29.670052+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.670052+0000', 'last_undegraded': '2024-03-21T08:42:29.670052+0000', 'last_fullsized': '2024-03-21T08:42:29.670052+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-03-21T08:42:35.640 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667552+0000', 'last_change': '2024-03-21T08:42:28.667552+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667552+0000', 'last_undegraded': '2024-03-21T08:42:28.667552+0000', 'last_fullsized': '2024-03-21T08:42:28.667552+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.673776+0000', 'last_change': '2024-03-21T08:42:29.673776+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.673776+0000', 'last_undegraded': '2024-03-21T08:42:29.673776+0000', 'last_fullsized': '2024-03-21T08:42:29.673776+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.663252+0000', 'last_change': '2024-03-21T08:42:28.663252+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.663252+0000', 'last_undegraded': '2024-03-21T08:42:28.663252+0000', 'last_fullsized': '2024-03-21T08:42:28.663252+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.663279+0000', 'last_change': '2024-03-21T08:42:28.663279+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.663279+0000', 'last_undegraded': '2024-03-21T08:42:28.663279+0000', 'last_fullsized': '2024-03-21T08:42:28.663279+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-03-21T08:42:35.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667313+0000', 'last_change': '2024-03-21T08:42:28.667313+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667313+0000', 'last_undegraded': '2024-03-21T08:42:28.667313+0000', 'last_fullsized': '2024-03-21T08:42:28.667313+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.666685+0000', 'last_change': '2024-03-21T08:42:28.666685+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.666685+0000', 'last_undegraded': '2024-03-21T08:42:28.666685+0000', 'last_fullsized': '2024-03-21T08:42:28.666685+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.677102+0000', 'last_change': '2024-03-21T08:42:29.677102+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.677102+0000', 'last_undegraded': '2024-03-21T08:42:29.677102+0000', 'last_fullsized': '2024-03-21T08:42:29.677102+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.642 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.663917+0000', 'last_change': '2024-03-21T08:42:28.663917+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.663917+0000', 'last_undegraded': '2024-03-21T08:42:28.663917+0000', 'last_fullsized': '2024-03-21T08:42:28.663917+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667605+0000', 'last_change': '2024-03-21T08:42:28.667605+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667605+0000', 'last_undegraded': '2024-03-21T08:42:28.667605+0000', 'last_fullsized': '2024-03-21T08:42:28.667605+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-03-21T08:42:35.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.669582+0000', 'last_change': '2024-03-21T08:42:29.669582+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.669582+0000', 'last_undegraded': '2024-03-21T08:42:29.669582+0000', 'last_fullsized': '2024-03-21T08:42:29.669582+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.668445+0000', 'last_change': '2024-03-21T08:42:29.668445+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.668445+0000', 'last_undegraded': '2024-03-21T08:42:29.668445+0000', 'last_fullsized': '2024-03-21T08:42:29.668445+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-03-21T08:42:35.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.671753+0000', 'last_change': '2024-03-21T08:42:29.671753+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.671753+0000', 'last_undegraded': '2024-03-21T08:42:29.671753+0000', 'last_fullsized': '2024-03-21T08:42:29.671753+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.663326+0000', 'last_change': '2024-03-21T08:42:28.663326+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.663326+0000', 'last_undegraded': '2024-03-21T08:42:28.663326+0000', 'last_fullsized': '2024-03-21T08:42:28.663326+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.673350+0000', 'last_change': '2024-03-21T08:42:29.673350+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.673350+0000', 'last_undegraded': '2024-03-21T08:42:29.673350+0000', 'last_fullsized': '2024-03-21T08:42:29.673350+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.667369+0000', 'last_change': '2024-03-21T08:42:28.667369+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.667369+0000', 'last_undegraded': '2024-03-21T08:42:28.667369+0000', 'last_fullsized': '2024-03-21T08:42:28.667369+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:42:35.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.666938+0000', 'last_change': '2024-03-21T08:42:28.666938+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.666938+0000', 'last_undegraded': '2024-03-21T08:42:28.666938+0000', 'last_fullsized': '2024-03-21T08:42:28.666938+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.672543+0000', 'last_change': '2024-03-21T08:42:29.672543+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.672543+0000', 'last_undegraded': '2024-03-21T08:42:29.672543+0000', 'last_fullsized': '2024-03-21T08:42:29.672543+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.666969+0000', 'last_change': '2024-03-21T08:42:28.666969+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.666969+0000', 'last_undegraded': '2024-03-21T08:42:28.666969+0000', 'last_fullsized': '2024-03-21T08:42:28.666969+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:42:35.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-03-21T08:42:35.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:29.675868+0000', 'last_change': '2024-03-21T08:42:29.675868+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.675868+0000', 'last_undegraded': '2024-03-21T08:42:29.675868+0000', 'last_fullsized': '2024-03-21T08:42:29.675868+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-21T08:42:35.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-03-21T08:42:35.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:42:28.666690+0000', 'last_change': '2024-03-21T08:42:28.666690+0000', 'last_active': '2024-03-21T08:42:28.653731+0000', 'last_peered': '2024-03-21T08:42:28.653731+0000', 'last_clean': '2024-03-21T08:42:28.653731+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:28.666690+0000', 'last_undegraded': '2024-03-21T08:42:28.666690+0000', 'last_fullsized': '2024-03-21T08:42:28.666690+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:28.653731+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:42:35.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-03-21T08:42:35.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-21T08:42:29.660854+0000', 'last_change': '2024-03-21T08:42:29.660854+0000', 'last_active': '2024-03-21T08:42:29.660854+0000', 'last_peered': '2024-03-21T08:42:29.660854+0000', 'last_clean': '2024-03-21T08:42:29.660854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:42:29.660854+0000', 'last_undegraded': '2024-03-21T08:42:29.660854+0000', 'last_fullsized': '2024-03-21T08:42:29.660854+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'last_clean_scrub_stamp': '2024-03-21T08:42:29.660854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-21T08:42:35.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-21T08:42:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.807+0000 7f97c5c00640 1 Processor -- start 2024-03-21T08:42:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.807+0000 7f97c5c00640 1 -- start start 2024-03-21T08:42:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c015dd00 0x7f97c00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97c00aff70 0x7f97c006b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 0x7f97c006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00d0540 con 0x7f97c00aff70 2024-03-21T08:42:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00cd6e0 con 0x7f97c006eb30 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97c5c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00cd9c0 con 0x7f97c015dd00 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97c00aff70 0x7f97c006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97c00aff70 0x7f97c006b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43198/0 (socket says 172.21.15.28:43198) 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 -- 172.21.15.28:0/570231731 learned_addr learned my addr 172.21.15.28:0/570231731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97bea00640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c015dd00 0x7f97c00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97bf400640 1 --2- 172.21.15.28:0/570231731 >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 0x7f97c006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 -- 172.21.15.28:0/570231731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c015dd00 msgr2=0x7f97c00cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c015dd00 0x7f97c00cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 -- 172.21.15.28:0/570231731 >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 msgr2=0x7f97c006b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 --2- 172.21.15.28:0/570231731 >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 0x7f97c006b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97be000640 1 -- 172.21.15.28:0/570231731 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00cdca0 con 0x7f97c00aff70 2024-03-21T08:42:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.811+0000 7f97bf400640 1 --2- 172.21.15.28:0/570231731 >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 0x7f97c006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97be000640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c00aff70 0x7f97c006b170 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f97c00c43b0 tx=0x7f97ac01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a77f4d268fd80d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97bd600640 1 -- 172.21.15.28:0/570231731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0aa430 con 0x7f97c00aff70 2024-03-21T08:42:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97bd600640 1 -- 172.21.15.28:0/570231731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac0af070 con 0x7f97c00aff70 2024-03-21T08:42:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97bd600640 1 -- 172.21.15.28:0/570231731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0aa7f0 con 0x7f97c00aff70 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- 172.21.15.28:0/570231731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c00aff70 msgr2=0x7f97c006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c00aff70 0x7f97c006b170 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f97c00c43b0 tx=0x7f97ac01c4f0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- 172.21.15.28:0/570231731 shutdown_connections 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- 172.21.15.28:0/570231731 >> v2:172.21.15.151:3300/0 conn(0x7f97c006eb30 0x7f97c006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c00aff70 0x7f97c006b170 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- 172.21.15.28:0/570231731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c015dd00 0x7f97c00cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- 172.21.15.28:0/570231731 >> 172.21.15.28:0/570231731 conn(0x7f97c0099a30 msgr2=0x7f97c0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- 172.21.15.28:0/570231731 shutdown_connections 2024-03-21T08:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- 172.21.15.28:0/570231731 wait complete. 2024-03-21T08:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 Processor -- start 2024-03-21T08:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 -- start start 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c00aff70 0x7f97c0151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.815+0000 7f97c5c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97c015dd00 0x7f97c0157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00d1400 con 0x7f97c006eb30 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00d1280 con 0x7f97c015dd00 2024-03-21T08:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00d1580 con 0x7f97c00aff70 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43206/0 (socket says 172.21.15.28:43206) 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 -- 172.21.15.28:0/2206837135 learned_addr learned my addr 172.21.15.28:0/2206837135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bf400640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97c015dd00 0x7f97c0157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97be000640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c00aff70 0x7f97c0151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 -- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c00aff70 msgr2=0x7f97c0151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c00aff70 0x7f97c0151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 -- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97c015dd00 msgr2=0x7f97c0157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97c015dd00 0x7f97c0157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 -- 172.21.15.28:0/2206837135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c01fb120 con 0x7f97c006eb30 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97bea00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f97b000d5f0 tx=0x7f97b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00125f0 con 0x7f97c006eb30 2024-03-21T08:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0019070 con 0x7f97c006eb30 2024-03-21T08:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c0150d60 con 0x7f97c006eb30 2024-03-21T08:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00129b0 con 0x7f97c006eb30 2024-03-21T08:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c0143080 con 0x7f97c006eb30 2024-03-21T08:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c0150b50 con 0x7f97c006eb30 2024-03-21T08:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.819+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f97b0012b50 con 0x7f97c006eb30 2024-03-21T08:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.823+0000 7f97b6a00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 0x7f97a0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.823+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97b00864e0 con 0x7f97c006eb30 2024-03-21T08:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.823+0000 7f97be000640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 0x7f97a0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.823+0000 7f97be000640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 0x7f97a0069fe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0ae000 tx=0x7f97ac0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:35.827+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f97b0056d50 con 0x7f97c006eb30 2024-03-21T08:42:36.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f97c004f410 con 0x7f97c006eb30 2024-03-21T08:42:36.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97b6a00640 1 -- 172.21.15.28:0/2206837135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1110 (secure 0 0 0) 0x7f97b0056650 con 0x7f97c006eb30 2024-03-21T08:42:36.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:36.049 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"51dba44a-cba1-4b57-a836-26a407968834","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":113,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1711010453,"num_in_osds":8,"osd_in_since":1711010445,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":44},{"state_name":"unknown","count":36},{"state_name":"active+clean","count":17}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":238575616,"bytes_avail":772855537664,"bytes_total":773094113280,"unknown_pgs_ratio":0.37113401293754578,"inactive_pgs_ratio":0.45360824465751648},"fsmap":{"epoch":18,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5517}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-21T08:42:13.732445+0000","services":{}},"progress_events":{}} 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 msgr2=0x7f97a0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 0x7f97a0069fe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0ae000 tx=0x7f97ac0a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 msgr2=0x7f97c0142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f97b000d5f0 tx=0x7f97b000dab0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 shutdown_connections 2024-03-21T08:42:36.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97c015dd00 0x7f97c0157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f97a0067b20 0x7f97a0069fe0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97c00aff70 0x7f97c0151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 --2- 172.21.15.28:0/2206837135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97c006eb30 0x7f97c0142b40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 >> 172.21.15.28:0/2206837135 conn(0x7f97c0099a30 msgr2=0x7f97c006b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 shutdown_connections 2024-03-21T08:42:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.047+0000 7f97c5c00640 1 -- 172.21.15.28:0/2206837135 wait complete. 2024-03-21T08:42:36.071 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-21T08:42:36.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:42:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 Processor -- start 2024-03-21T08:42:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 -- start start 2024-03-21T08:42:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9ec009af80 0x7f9ec0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec00d0780 con 0x7f9ec009a0c0 2024-03-21T08:42:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec00cc810 con 0x7f9ec015d060 2024-03-21T08:42:36.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec7a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec009a510 con 0x7f9ec009af80 2024-03-21T08:42:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec5a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9ec009a0c0 0x7f9ec00cd2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54512/0 (socket says 172.21.15.28:54512) 2024-03-21T08:42:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ec5a00640 1 -- 172.21.15.28:0/3226365562 learned_addr learned my addr 172.21.15.28:0/3226365562 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9eb7e00640 1 --2- 172.21.15.28:0/3226365562 >> v2:172.21.15.151:3300/0 conn(0x7f9ec009af80 0x7f9ec0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774566576 0 0) 0x7f9ec00d0780 con 0x7f9ec009a0c0 2024-03-21T08:42:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea4003530 con 0x7f9ec009a0c0 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469494778 0 0) 0x7f9ec00cc810 con 0x7f9ec015d060 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec00d0780 con 0x7f9ec015d060 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9eb7e00640 1 -- 172.21.15.28:0/3226365562 >> v1:172.21.15.151:6789/0 conn(0x7f9ec015d060 legacy=0x7f9ec00b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854072715 0 0) 0x7f9ea4003530 con 0x7f9ec009a0c0 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9eb7e00640 1 -- 172.21.15.28:0/3226365562 >> v1:172.21.15.28:6789/0 conn(0x7f9ec009a0c0 legacy=0x7f9ec00cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.211+0000 7f9eb7e00640 1 -- 172.21.15.28:0/3226365562 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00ccaf0 con 0x7f9ec009af80 2024-03-21T08:42:36.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9eb7e00640 1 --2- 172.21.15.28:0/3226365562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec0099b80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80097c0 tx=0x7f9ea801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ec2c7b1707febca server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:36.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea80b0070 con 0x7f9ec009af80 2024-03-21T08:42:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea80ab400 con 0x7f9ec009af80 2024-03-21T08:42:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ebfe00640 1 -- 172.21.15.28:0/3226365562 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8001a00 con 0x7f9ec009af80 2024-03-21T08:42:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 -- 172.21.15.28:0/3226365562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 msgr2=0x7f9ec0099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/3226365562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec0099b80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80097c0 tx=0x7f9ea801c3c0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 -- 172.21.15.28:0/3226365562 shutdown_connections 2024-03-21T08:42:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/3226365562 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec0099b80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 -- 172.21.15.28:0/3226365562 >> 172.21.15.28:0/3226365562 conn(0x7f9ec00b3510 msgr2=0x7f9ec00b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.215+0000 7f9ec7a00640 1 -- 172.21.15.28:0/3226365562 shutdown_connections 2024-03-21T08:42:36.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 -- 172.21.15.28:0/3226365562 wait complete. 2024-03-21T08:42:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 Processor -- start 2024-03-21T08:42:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 -- start start 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ec009a0c0 0x7f9ec006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec01538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00b09f0 con 0x7f9ec015d060 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00d0a60 con 0x7f9ec009af80 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec7a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec00b0b70 con 0x7f9ec009a0c0 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43210/0 (socket says 172.21.15.28:43210) 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 -- 172.21.15.28:0/1513400869 learned_addr learned my addr 172.21.15.28:0/1513400869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9eb7e00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec01538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5000640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ec009a0c0 0x7f9ec006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 -- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ec009a0c0 msgr2=0x7f9ec006deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ec009a0c0 0x7f9ec006deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 -- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 msgr2=0x7f9ec01538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec01538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 -- 172.21.15.28:0/1513400869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec0156530 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.219+0000 7f9ec5a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800cc30 tx=0x7f9eb800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8017070 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb800ce40 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8012620 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec0157150 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0158c30 con 0x7f9ec015d060 2024-03-21T08:42:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec004f410 con 0x7f9ec015d060 2024-03-21T08:42:36.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.223+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9eb801a070 con 0x7f9ec015d060 2024-03-21T08:42:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.227+0000 7f9ebd600640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 0x7f9e9c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.227+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9eb8085fd0 con 0x7f9ec015d060 2024-03-21T08:42:36.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.227+0000 7f9ec5000640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 0x7f9e9c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:36.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.227+0000 7f9ec5000640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 0x7f9e9c069fe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00011f0 tx=0x7f9eb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:36.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.231+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9ec0157150 con 0x7f9ec015d060 2024-03-21T08:42:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.391+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ec0156700 con 0x7f9e9c067b20 2024-03-21T08:42:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.399+0000 7f9ebd600640 1 -- 172.21.15.28:0/1513400869 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7f9ec0156700 con 0x7f9e9c067b20 2024-03-21T08:42:36.402 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 msgr2=0x7f9e9c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 0x7f9e9c069fe0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00011f0 tx=0x7f9eb0009040 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 msgr2=0x7f9ec00af850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800cc30 tx=0x7f9eb800a430 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 shutdown_connections 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ec015d060 0x7f9ec00af850 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9ec009af80 0x7f9ec01538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e9c067b20 0x7f9e9c069fe0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 --2- 172.21.15.28:0/1513400869 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9ec009a0c0 0x7f9ec006deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 >> 172.21.15.28:0/1513400869 conn(0x7f9ec00b3510 msgr2=0x7f9ec015abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 shutdown_connections 2024-03-21T08:42:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:36.403+0000 7f9ec7a00640 1 -- 172.21.15.28:0/1513400869 wait complete. 2024-03-21T08:42:36.423 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-03-21T08:42:35.737032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":161,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237516,"kb_used_data":4680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737204,"statfs":{"total":773094113280,"available":772850896896,"internally_reserved":0,"allocated":4792320,"data_stored":2721724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.072017"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046862+0000","last_change":"2024-03-21T08:42:30.695871+0000","last_active":"2024-03-21T08:42:32.046862+0000","last_peered":"2024-03-21T08:42:32.046862+0000","last_clean":"2024-03-21T08:42:32.046862+0000","last_became_active":"2024-03-21T08:42:30.695549+0000","last_became_peered":"2024-03-21T08:42:30.695549+0000","last_unstale":"2024-03-21T08:42:32.046862+0000","last_undegraded":"2024-03-21T08:42:32.046862+0000","last_fullsized":"2024-03-21T08:42:32.046862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:50:31.978070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046492+0000","last_change":"2024-03-21T08:42:30.687207+0000","last_active":"2024-03-21T08:42:32.046492+0000","last_peered":"2024-03-21T08:42:32.046492+0000","last_clean":"2024-03-21T08:42:32.046492+0000","last_became_active":"2024-03-21T08:42:30.686933+0000","last_became_peered":"2024-03-21T08:42:30.686933+0000","last_unstale":"2024-03-21T08:42:32.046492+0000","last_undegraded":"2024-03-21T08:42:32.046492+0000","last_fullsized":"2024-03-21T08:42:32.046492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:40.800907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751168+0000","last_change":"2024-03-21T08:42:30.682877+0000","last_active":"2024-03-21T08:42:31.751168+0000","last_peered":"2024-03-21T08:42:31.751168+0000","last_clean":"2024-03-21T08:42:31.751168+0000","last_became_active":"2024-03-21T08:42:30.682612+0000","last_became_peered":"2024-03-21T08:42:30.682612+0000","last_unstale":"2024-03-21T08:42:31.751168+0000","last_undegraded":"2024-03-21T08:42:31.751168+0000","last_fullsized":"2024-03-21T08:42:31.751168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:12.042436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686840+0000","last_change":"2024-03-21T08:42:30.678176+0000","last_active":"2024-03-21T08:42:31.686840+0000","last_peered":"2024-03-21T08:42:31.686840+0000","last_clean":"2024-03-21T08:42:31.686840+0000","last_became_active":"2024-03-21T08:42:30.677904+0000","last_became_peered":"2024-03-21T08:42:30.677904+0000","last_unstale":"2024-03-21T08:42:31.686840+0000","last_undegraded":"2024-03-21T08:42:31.686840+0000","last_fullsized":"2024-03-21T08:42:31.686840+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:55.306648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686403+0000","last_change":"2024-03-21T08:42:30.677271+0000","last_active":"2024-03-21T08:42:31.686403+0000","last_peered":"2024-03-21T08:42:31.686403+0000","last_clean":"2024-03-21T08:42:31.686403+0000","last_became_active":"2024-03-21T08:42:30.676973+0000","last_became_peered":"2024-03-21T08:42:30.676973+0000","last_unstale":"2024-03-21T08:42:31.686403+0000","last_undegraded":"2024-03-21T08:42:31.686403+0000","last_fullsized":"2024-03-21T08:42:31.686403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:57.865608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046416+0000","last_change":"2024-03-21T08:42:30.686840+0000","last_active":"2024-03-21T08:42:32.046416+0000","last_peered":"2024-03-21T08:42:32.046416+0000","last_clean":"2024-03-21T08:42:32.046416+0000","last_became_active":"2024-03-21T08:42:30.686561+0000","last_became_peered":"2024-03-21T08:42:30.686561+0000","last_unstale":"2024-03-21T08:42:32.046416+0000","last_undegraded":"2024-03-21T08:42:32.046416+0000","last_fullsized":"2024-03-21T08:42:32.046416+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:55:42.547751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687096+0000","last_change":"2024-03-21T08:42:30.678888+0000","last_active":"2024-03-21T08:42:31.687096+0000","last_peered":"2024-03-21T08:42:31.687096+0000","last_clean":"2024-03-21T08:42:31.687096+0000","last_became_active":"2024-03-21T08:42:30.678604+0000","last_became_peered":"2024-03-21T08:42:30.678604+0000","last_unstale":"2024-03-21T08:42:31.687096+0000","last_undegraded":"2024-03-21T08:42:31.687096+0000","last_fullsized":"2024-03-21T08:42:31.687096+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:11:21.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046263+0000","last_change":"2024-03-21T08:42:30.691873+0000","last_active":"2024-03-21T08:42:32.046263+0000","last_peered":"2024-03-21T08:42:32.046263+0000","last_clean":"2024-03-21T08:42:32.046263+0000","last_became_active":"2024-03-21T08:42:30.687997+0000","last_became_peered":"2024-03-21T08:42:30.687997+0000","last_unstale":"2024-03-21T08:42:32.046263+0000","last_undegraded":"2024-03-21T08:42:32.046263+0000","last_fullsized":"2024-03-21T08:42:32.046263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:24.326977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751237+0000","last_change":"2024-03-21T08:42:30.685358+0000","last_active":"2024-03-21T08:42:31.751237+0000","last_peered":"2024-03-21T08:42:31.751237+0000","last_clean":"2024-03-21T08:42:31.751237+0000","last_became_active":"2024-03-21T08:42:30.685071+0000","last_became_peered":"2024-03-21T08:42:30.685071+0000","last_unstale":"2024-03-21T08:42:31.751237+0000","last_undegraded":"2024-03-21T08:42:31.751237+0000","last_fullsized":"2024-03-21T08:42:31.751237+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:04:04.312171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687195+0000","last_change":"2024-03-21T08:42:30.695493+0000","last_active":"2024-03-21T08:42:31.687195+0000","last_peered":"2024-03-21T08:42:31.687195+0000","last_clean":"2024-03-21T08:42:31.687195+0000","last_became_active":"2024-03-21T08:42:30.695195+0000","last_became_peered":"2024-03-21T08:42:30.695195+0000","last_unstale":"2024-03-21T08:42:31.687195+0000","last_undegraded":"2024-03-21T08:42:31.687195+0000","last_fullsized":"2024-03-21T08:42:31.687195+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:58:57.974782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687736+0000","last_change":"2024-03-21T08:42:30.688105+0000","last_active":"2024-03-21T08:42:31.687736+0000","last_peered":"2024-03-21T08:42:31.687736+0000","last_clean":"2024-03-21T08:42:31.687736+0000","last_became_active":"2024-03-21T08:42:30.686104+0000","last_became_peered":"2024-03-21T08:42:30.686104+0000","last_unstale":"2024-03-21T08:42:31.687736+0000","last_undegraded":"2024-03-21T08:42:31.687736+0000","last_fullsized":"2024-03-21T08:42:31.687736+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:38:41.370318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750802+0000","last_change":"2024-03-21T08:42:30.689408+0000","last_active":"2024-03-21T08:42:31.750802+0000","last_peered":"2024-03-21T08:42:31.750802+0000","last_clean":"2024-03-21T08:42:31.750802+0000","last_became_active":"2024-03-21T08:42:30.689100+0000","last_became_peered":"2024-03-21T08:42:30.689100+0000","last_unstale":"2024-03-21T08:42:31.750802+0000","last_undegraded":"2024-03-21T08:42:31.750802+0000","last_fullsized":"2024-03-21T08:42:31.750802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:21.214239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687225+0000","last_change":"2024-03-21T08:42:30.698030+0000","last_active":"2024-03-21T08:42:31.687225+0000","last_peered":"2024-03-21T08:42:31.687225+0000","last_clean":"2024-03-21T08:42:31.687225+0000","last_became_active":"2024-03-21T08:42:30.697796+0000","last_became_peered":"2024-03-21T08:42:30.697796+0000","last_unstale":"2024-03-21T08:42:31.687225+0000","last_undegraded":"2024-03-21T08:42:31.687225+0000","last_fullsized":"2024-03-21T08:42:31.687225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:32.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046166+0000","last_change":"2024-03-21T08:42:30.687179+0000","last_active":"2024-03-21T08:42:32.046166+0000","last_peered":"2024-03-21T08:42:32.046166+0000","last_clean":"2024-03-21T08:42:32.046166+0000","last_became_active":"2024-03-21T08:42:30.686870+0000","last_became_peered":"2024-03-21T08:42:30.686870+0000","last_unstale":"2024-03-21T08:42:32.046166+0000","last_undegraded":"2024-03-21T08:42:32.046166+0000","last_fullsized":"2024-03-21T08:42:32.046166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:03:40.989778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686963+0000","last_change":"2024-03-21T08:42:30.686337+0000","last_active":"2024-03-21T08:42:31.686963+0000","last_peered":"2024-03-21T08:42:31.686963+0000","last_clean":"2024-03-21T08:42:31.686963+0000","last_became_active":"2024-03-21T08:42:30.686026+0000","last_became_peered":"2024-03-21T08:42:30.686026+0000","last_unstale":"2024-03-21T08:42:31.686963+0000","last_undegraded":"2024-03-21T08:42:31.686963+0000","last_fullsized":"2024-03-21T08:42:31.686963+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:24.326977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686779+0000","last_change":"2024-03-21T08:42:30.695084+0000","last_active":"2024-03-21T08:42:31.686779+0000","last_peered":"2024-03-21T08:42:31.686779+0000","last_clean":"2024-03-21T08:42:31.686779+0000","last_became_active":"2024-03-21T08:42:30.694799+0000","last_became_peered":"2024-03-21T08:42:30.694799+0000","last_unstale":"2024-03-21T08:42:31.686779+0000","last_undegraded":"2024-03-21T08:42:31.686779+0000","last_fullsized":"2024-03-21T08:42:31.686779+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:29:30.921384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687929+0000","last_change":"2024-03-21T08:42:30.700752+0000","last_active":"2024-03-21T08:42:31.687929+0000","last_peered":"2024-03-21T08:42:31.687929+0000","last_clean":"2024-03-21T08:42:31.687929+0000","last_became_active":"2024-03-21T08:42:30.698996+0000","last_became_peered":"2024-03-21T08:42:30.698996+0000","last_unstale":"2024-03-21T08:42:31.687929+0000","last_undegraded":"2024-03-21T08:42:31.687929+0000","last_fullsized":"2024-03-21T08:42:31.687929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:16:19.492004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046260+0000","last_change":"2024-03-21T08:42:30.694366+0000","last_active":"2024-03-21T08:42:32.046260+0000","last_peered":"2024-03-21T08:42:32.046260+0000","last_clean":"2024-03-21T08:42:32.046260+0000","last_became_active":"2024-03-21T08:42:30.693997+0000","last_became_peered":"2024-03-21T08:42:30.693997+0000","last_unstale":"2024-03-21T08:42:32.046260+0000","last_undegraded":"2024-03-21T08:42:32.046260+0000","last_fullsized":"2024-03-21T08:42:32.046260+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:06:29.004037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686922+0000","last_change":"2024-03-21T08:42:30.691546+0000","last_active":"2024-03-21T08:42:31.686922+0000","last_peered":"2024-03-21T08:42:31.686922+0000","last_clean":"2024-03-21T08:42:31.686922+0000","last_became_active":"2024-03-21T08:42:30.691237+0000","last_became_peered":"2024-03-21T08:42:30.691237+0000","last_unstale":"2024-03-21T08:42:31.686922+0000","last_undegraded":"2024-03-21T08:42:31.686922+0000","last_fullsized":"2024-03-21T08:42:31.686922+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:04.901041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046219+0000","last_change":"2024-03-21T08:42:30.698276+0000","last_active":"2024-03-21T08:42:32.046219+0000","last_peered":"2024-03-21T08:42:32.046219+0000","last_clean":"2024-03-21T08:42:32.046219+0000","last_became_active":"2024-03-21T08:42:30.695309+0000","last_became_peered":"2024-03-21T08:42:30.695309+0000","last_unstale":"2024-03-21T08:42:32.046219+0000","last_undegraded":"2024-03-21T08:42:32.046219+0000","last_fullsized":"2024-03-21T08:42:32.046219+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:03.053281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687785+0000","last_change":"2024-03-21T08:42:30.694593+0000","last_active":"2024-03-21T08:42:31.687785+0000","last_peered":"2024-03-21T08:42:31.687785+0000","last_clean":"2024-03-21T08:42:31.687785+0000","last_became_active":"2024-03-21T08:42:30.694289+0000","last_became_peered":"2024-03-21T08:42:30.694289+0000","last_unstale":"2024-03-21T08:42:31.687785+0000","last_undegraded":"2024-03-21T08:42:31.687785+0000","last_fullsized":"2024-03-21T08:42:31.687785+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046761+0000","last_change":"2024-03-21T08:42:30.687038+0000","last_active":"2024-03-21T08:42:32.046761+0000","last_peered":"2024-03-21T08:42:32.046761+0000","last_clean":"2024-03-21T08:42:32.046761+0000","last_became_active":"2024-03-21T08:42:30.686689+0000","last_became_peered":"2024-03-21T08:42:30.686689+0000","last_unstale":"2024-03-21T08:42:32.046761+0000","last_undegraded":"2024-03-21T08:42:32.046761+0000","last_fullsized":"2024-03-21T08:42:32.046761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:51:06.301613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688618+0000","last_change":"2024-03-21T08:42:30.696890+0000","last_active":"2024-03-21T08:42:31.688618+0000","last_peered":"2024-03-21T08:42:31.688618+0000","last_clean":"2024-03-21T08:42:31.688618+0000","last_became_active":"2024-03-21T08:42:30.696103+0000","last_became_peered":"2024-03-21T08:42:30.696103+0000","last_unstale":"2024-03-21T08:42:31.688618+0000","last_undegraded":"2024-03-21T08:42:31.688618+0000","last_fullsized":"2024-03-21T08:42:31.688618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:51:38.815895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686521+0000","last_change":"2024-03-21T08:42:30.695584+0000","last_active":"2024-03-21T08:42:31.686521+0000","last_peered":"2024-03-21T08:42:31.686521+0000","last_clean":"2024-03-21T08:42:31.686521+0000","last_became_active":"2024-03-21T08:42:30.695327+0000","last_became_peered":"2024-03-21T08:42:30.695327+0000","last_unstale":"2024-03-21T08:42:31.686521+0000","last_undegraded":"2024-03-21T08:42:31.686521+0000","last_fullsized":"2024-03-21T08:42:31.686521+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:14.494767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046526+0000","last_change":"2024-03-21T08:42:30.694013+0000","last_active":"2024-03-21T08:42:32.046526+0000","last_peered":"2024-03-21T08:42:32.046526+0000","last_clean":"2024-03-21T08:42:32.046526+0000","last_became_active":"2024-03-21T08:42:30.693709+0000","last_became_peered":"2024-03-21T08:42:30.693709+0000","last_unstale":"2024-03-21T08:42:32.046526+0000","last_undegraded":"2024-03-21T08:42:32.046526+0000","last_fullsized":"2024-03-21T08:42:32.046526+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:22:22.827282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047041+0000","last_change":"2024-03-21T08:42:30.693187+0000","last_active":"2024-03-21T08:42:32.047041+0000","last_peered":"2024-03-21T08:42:32.047041+0000","last_clean":"2024-03-21T08:42:32.047041+0000","last_became_active":"2024-03-21T08:42:30.690678+0000","last_became_peered":"2024-03-21T08:42:30.690678+0000","last_unstale":"2024-03-21T08:42:32.047041+0000","last_undegraded":"2024-03-21T08:42:32.047041+0000","last_fullsized":"2024-03-21T08:42:32.047041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:29.930403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046144+0000","last_change":"2024-03-21T08:42:30.694219+0000","last_active":"2024-03-21T08:42:32.046144+0000","last_peered":"2024-03-21T08:42:32.046144+0000","last_clean":"2024-03-21T08:42:32.046144+0000","last_became_active":"2024-03-21T08:42:30.688701+0000","last_became_peered":"2024-03-21T08:42:30.688701+0000","last_unstale":"2024-03-21T08:42:32.046144+0000","last_undegraded":"2024-03-21T08:42:32.046144+0000","last_fullsized":"2024-03-21T08:42:32.046144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:30:56.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688494+0000","last_change":"2024-03-21T08:42:30.690686+0000","last_active":"2024-03-21T08:42:31.688494+0000","last_peered":"2024-03-21T08:42:31.688494+0000","last_clean":"2024-03-21T08:42:31.688494+0000","last_became_active":"2024-03-21T08:42:30.690472+0000","last_became_peered":"2024-03-21T08:42:30.690472+0000","last_unstale":"2024-03-21T08:42:31.688494+0000","last_undegraded":"2024-03-21T08:42:31.688494+0000","last_fullsized":"2024-03-21T08:42:31.688494+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:50:31.978070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046068+0000","last_change":"2024-03-21T08:42:30.693396+0000","last_active":"2024-03-21T08:42:32.046068+0000","last_peered":"2024-03-21T08:42:32.046068+0000","last_clean":"2024-03-21T08:42:32.046068+0000","last_became_active":"2024-03-21T08:42:30.693136+0000","last_became_peered":"2024-03-21T08:42:30.693136+0000","last_unstale":"2024-03-21T08:42:32.046068+0000","last_undegraded":"2024-03-21T08:42:32.046068+0000","last_fullsized":"2024-03-21T08:42:32.046068+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:28.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045777+0000","last_change":"2024-03-21T08:42:30.691385+0000","last_active":"2024-03-21T08:42:32.045777+0000","last_peered":"2024-03-21T08:42:32.045777+0000","last_clean":"2024-03-21T08:42:32.045777+0000","last_became_active":"2024-03-21T08:42:30.690817+0000","last_became_peered":"2024-03-21T08:42:30.690817+0000","last_unstale":"2024-03-21T08:42:32.045777+0000","last_undegraded":"2024-03-21T08:42:32.045777+0000","last_fullsized":"2024-03-21T08:42:32.045777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:16:04.376219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688690+0000","last_change":"2024-03-21T08:42:30.696929+0000","last_active":"2024-03-21T08:42:31.688690+0000","last_peered":"2024-03-21T08:42:31.688690+0000","last_clean":"2024-03-21T08:42:31.688690+0000","last_became_active":"2024-03-21T08:42:30.696470+0000","last_became_peered":"2024-03-21T08:42:30.696470+0000","last_unstale":"2024-03-21T08:42:31.688690+0000","last_undegraded":"2024-03-21T08:42:31.688690+0000","last_fullsized":"2024-03-21T08:42:31.688690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:41:08.879588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046063+0000","last_change":"2024-03-21T08:42:30.694122+0000","last_active":"2024-03-21T08:42:32.046063+0000","last_peered":"2024-03-21T08:42:32.046063+0000","last_clean":"2024-03-21T08:42:32.046063+0000","last_became_active":"2024-03-21T08:42:30.691718+0000","last_became_peered":"2024-03-21T08:42:30.691718+0000","last_unstale":"2024-03-21T08:42:32.046063+0000","last_undegraded":"2024-03-21T08:42:32.046063+0000","last_fullsized":"2024-03-21T08:42:32.046063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:16.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688224+0000","last_change":"2024-03-21T08:42:30.692105+0000","last_active":"2024-03-21T08:42:31.688224+0000","last_peered":"2024-03-21T08:42:31.688224+0000","last_clean":"2024-03-21T08:42:31.688224+0000","last_became_active":"2024-03-21T08:42:30.691865+0000","last_became_peered":"2024-03-21T08:42:30.691865+0000","last_unstale":"2024-03-21T08:42:31.688224+0000","last_undegraded":"2024-03-21T08:42:31.688224+0000","last_fullsized":"2024-03-21T08:42:31.688224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:05.453862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046636+0000","last_change":"2024-03-21T08:42:29.683944+0000","last_active":"2024-03-21T08:42:32.046636+0000","last_peered":"2024-03-21T08:42:32.046636+0000","last_clean":"2024-03-21T08:42:32.046636+0000","last_became_active":"2024-03-21T08:42:29.683089+0000","last_became_peered":"2024-03-21T08:42:29.683089+0000","last_unstale":"2024-03-21T08:42:32.046636+0000","last_undegraded":"2024-03-21T08:42:32.046636+0000","last_fullsized":"2024-03-21T08:42:32.046636+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047155+0000","last_change":"2024-03-21T08:42:30.700542+0000","last_active":"2024-03-21T08:42:32.047155+0000","last_peered":"2024-03-21T08:42:32.047155+0000","last_clean":"2024-03-21T08:42:32.047155+0000","last_became_active":"2024-03-21T08:42:30.696121+0000","last_became_peered":"2024-03-21T08:42:30.696121+0000","last_unstale":"2024-03-21T08:42:32.047155+0000","last_undegraded":"2024-03-21T08:42:32.047155+0000","last_fullsized":"2024-03-21T08:42:32.047155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:29:30.921384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750995+0000","last_change":"2024-03-21T08:42:29.688290+0000","last_active":"2024-03-21T08:42:31.750995+0000","last_peered":"2024-03-21T08:42:31.750995+0000","last_clean":"2024-03-21T08:42:31.750995+0000","last_became_active":"2024-03-21T08:42:29.687997+0000","last_became_peered":"2024-03-21T08:42:29.687997+0000","last_unstale":"2024-03-21T08:42:31.750995+0000","last_undegraded":"2024-03-21T08:42:31.750995+0000","last_fullsized":"2024-03-21T08:42:31.750995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:11.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046694+0000","last_change":"2024-03-21T08:42:30.692448+0000","last_active":"2024-03-21T08:42:32.046694+0000","last_peered":"2024-03-21T08:42:32.046694+0000","last_clean":"2024-03-21T08:42:32.046694+0000","last_became_active":"2024-03-21T08:42:30.692149+0000","last_became_peered":"2024-03-21T08:42:30.692149+0000","last_unstale":"2024-03-21T08:42:32.046694+0000","last_undegraded":"2024-03-21T08:42:32.046694+0000","last_fullsized":"2024-03-21T08:42:32.046694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:55:15.669183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047168+0000","last_change":"2024-03-21T08:42:29.683368+0000","last_active":"2024-03-21T08:42:32.047168+0000","last_peered":"2024-03-21T08:42:32.047168+0000","last_clean":"2024-03-21T08:42:32.047168+0000","last_became_active":"2024-03-21T08:42:29.683095+0000","last_became_peered":"2024-03-21T08:42:29.683095+0000","last_unstale":"2024-03-21T08:42:32.047168+0000","last_undegraded":"2024-03-21T08:42:32.047168+0000","last_fullsized":"2024-03-21T08:42:32.047168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:38.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046766+0000","last_change":"2024-03-21T08:42:30.704171+0000","last_active":"2024-03-21T08:42:32.046766+0000","last_peered":"2024-03-21T08:42:32.046766+0000","last_clean":"2024-03-21T08:42:32.046766+0000","last_became_active":"2024-03-21T08:42:30.703815+0000","last_became_peered":"2024-03-21T08:42:30.703815+0000","last_unstale":"2024-03-21T08:42:32.046766+0000","last_undegraded":"2024-03-21T08:42:32.046766+0000","last_fullsized":"2024-03-21T08:42:32.046766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:38:41.370318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046530+0000","last_change":"2024-03-21T08:42:29.687300+0000","last_active":"2024-03-21T08:42:32.046530+0000","last_peered":"2024-03-21T08:42:32.046530+0000","last_clean":"2024-03-21T08:42:32.046530+0000","last_became_active":"2024-03-21T08:42:29.687010+0000","last_became_peered":"2024-03-21T08:42:29.687010+0000","last_unstale":"2024-03-21T08:42:32.046530+0000","last_undegraded":"2024-03-21T08:42:32.046530+0000","last_fullsized":"2024-03-21T08:42:32.046530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:11.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750892+0000","last_change":"2024-03-21T08:42:30.692975+0000","last_active":"2024-03-21T08:42:31.750892+0000","last_peered":"2024-03-21T08:42:31.750892+0000","last_clean":"2024-03-21T08:42:31.750892+0000","last_became_active":"2024-03-21T08:42:30.692636+0000","last_became_peered":"2024-03-21T08:42:30.692636+0000","last_unstale":"2024-03-21T08:42:31.750892+0000","last_undegraded":"2024-03-21T08:42:31.750892+0000","last_fullsized":"2024-03-21T08:42:31.750892+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:55:15.669183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687279+0000","last_change":"2024-03-21T08:42:29.680829+0000","last_active":"2024-03-21T08:42:31.687279+0000","last_peered":"2024-03-21T08:42:31.687279+0000","last_clean":"2024-03-21T08:42:31.687279+0000","last_became_active":"2024-03-21T08:42:29.680539+0000","last_became_peered":"2024-03-21T08:42:29.680539+0000","last_unstale":"2024-03-21T08:42:31.687279+0000","last_undegraded":"2024-03-21T08:42:31.687279+0000","last_fullsized":"2024-03-21T08:42:31.687279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:45.341543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687157+0000","last_change":"2024-03-21T08:42:30.693765+0000","last_active":"2024-03-21T08:42:31.687157+0000","last_peered":"2024-03-21T08:42:31.687157+0000","last_clean":"2024-03-21T08:42:31.687157+0000","last_became_active":"2024-03-21T08:42:30.693460+0000","last_became_peered":"2024-03-21T08:42:30.693460+0000","last_unstale":"2024-03-21T08:42:31.687157+0000","last_undegraded":"2024-03-21T08:42:31.687157+0000","last_fullsized":"2024-03-21T08:42:31.687157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:40.800907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687619+0000","last_change":"2024-03-21T08:42:29.689467+0000","last_active":"2024-03-21T08:42:31.687619+0000","last_peered":"2024-03-21T08:42:31.687619+0000","last_clean":"2024-03-21T08:42:31.687619+0000","last_became_active":"2024-03-21T08:42:29.688797+0000","last_became_peered":"2024-03-21T08:42:29.688797+0000","last_unstale":"2024-03-21T08:42:31.687619+0000","last_undegraded":"2024-03-21T08:42:31.687619+0000","last_fullsized":"2024-03-21T08:42:31.687619+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:38:24.494075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046343+0000","last_change":"2024-03-21T08:42:30.680880+0000","last_active":"2024-03-21T08:42:32.046343+0000","last_peered":"2024-03-21T08:42:32.046343+0000","last_clean":"2024-03-21T08:42:32.046343+0000","last_became_active":"2024-03-21T08:42:30.680614+0000","last_became_peered":"2024-03-21T08:42:30.680614+0000","last_unstale":"2024-03-21T08:42:32.046343+0000","last_undegraded":"2024-03-21T08:42:32.046343+0000","last_fullsized":"2024-03-21T08:42:32.046343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:33:36.238601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046141+0000","last_change":"2024-03-21T08:42:29.688043+0000","last_active":"2024-03-21T08:42:32.046141+0000","last_peered":"2024-03-21T08:42:32.046141+0000","last_clean":"2024-03-21T08:42:32.046141+0000","last_became_active":"2024-03-21T08:42:29.686991+0000","last_became_peered":"2024-03-21T08:42:29.686991+0000","last_unstale":"2024-03-21T08:42:32.046141+0000","last_undegraded":"2024-03-21T08:42:32.046141+0000","last_fullsized":"2024-03-21T08:42:32.046141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:03.893918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046640+0000","last_change":"2024-03-21T08:42:30.702252+0000","last_active":"2024-03-21T08:42:32.046640+0000","last_peered":"2024-03-21T08:42:32.046640+0000","last_clean":"2024-03-21T08:42:32.046640+0000","last_became_active":"2024-03-21T08:42:30.701982+0000","last_became_peered":"2024-03-21T08:42:30.701982+0000","last_unstale":"2024-03-21T08:42:32.046640+0000","last_undegraded":"2024-03-21T08:42:32.046640+0000","last_fullsized":"2024-03-21T08:42:32.046640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:28:41.046722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045717+0000","last_change":"2024-03-21T08:42:29.688835+0000","last_active":"2024-03-21T08:42:32.045717+0000","last_peered":"2024-03-21T08:42:32.045717+0000","last_clean":"2024-03-21T08:42:32.045717+0000","last_became_active":"2024-03-21T08:42:29.688547+0000","last_became_peered":"2024-03-21T08:42:29.688547+0000","last_unstale":"2024-03-21T08:42:32.045717+0000","last_undegraded":"2024-03-21T08:42:32.045717+0000","last_fullsized":"2024-03-21T08:42:32.045717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:00:31.687764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687081+0000","last_change":"2024-03-21T08:42:30.678062+0000","last_active":"2024-03-21T08:42:31.687081+0000","last_peered":"2024-03-21T08:42:31.687081+0000","last_clean":"2024-03-21T08:42:31.687081+0000","last_became_active":"2024-03-21T08:42:30.676930+0000","last_became_peered":"2024-03-21T08:42:30.676930+0000","last_unstale":"2024-03-21T08:42:31.687081+0000","last_undegraded":"2024-03-21T08:42:31.687081+0000","last_fullsized":"2024-03-21T08:42:31.687081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:39.027444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046818+0000","last_change":"2024-03-21T08:42:29.689147+0000","last_active":"2024-03-21T08:42:32.046818+0000","last_peered":"2024-03-21T08:42:32.046818+0000","last_clean":"2024-03-21T08:42:32.046818+0000","last_became_active":"2024-03-21T08:42:29.688858+0000","last_became_peered":"2024-03-21T08:42:29.688858+0000","last_unstale":"2024-03-21T08:42:32.046818+0000","last_undegraded":"2024-03-21T08:42:32.046818+0000","last_fullsized":"2024-03-21T08:42:32.046818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:04:54.658152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688049+0000","last_change":"2024-03-21T08:42:30.698953+0000","last_active":"2024-03-21T08:42:31.688049+0000","last_peered":"2024-03-21T08:42:31.688049+0000","last_clean":"2024-03-21T08:42:31.688049+0000","last_became_active":"2024-03-21T08:42:30.698585+0000","last_became_peered":"2024-03-21T08:42:30.698585+0000","last_unstale":"2024-03-21T08:42:31.688049+0000","last_undegraded":"2024-03-21T08:42:31.688049+0000","last_fullsized":"2024-03-21T08:42:31.688049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:30.269037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687406+0000","last_change":"2024-03-21T08:42:29.690665+0000","last_active":"2024-03-21T08:42:31.687406+0000","last_peered":"2024-03-21T08:42:31.687406+0000","last_clean":"2024-03-21T08:42:31.687406+0000","last_became_active":"2024-03-21T08:42:29.690396+0000","last_became_peered":"2024-03-21T08:42:29.690396+0000","last_unstale":"2024-03-21T08:42:31.687406+0000","last_undegraded":"2024-03-21T08:42:31.687406+0000","last_fullsized":"2024-03-21T08:42:31.687406+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:15.673658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751255+0000","last_change":"2024-03-21T08:42:30.697681+0000","last_active":"2024-03-21T08:42:31.751255+0000","last_peered":"2024-03-21T08:42:31.751255+0000","last_clean":"2024-03-21T08:42:31.751255+0000","last_became_active":"2024-03-21T08:42:30.697400+0000","last_became_peered":"2024-03-21T08:42:30.697400+0000","last_unstale":"2024-03-21T08:42:31.751255+0000","last_undegraded":"2024-03-21T08:42:31.751255+0000","last_fullsized":"2024-03-21T08:42:31.751255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:32:42.990151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687899+0000","last_change":"2024-03-21T08:42:29.681244+0000","last_active":"2024-03-21T08:42:31.687899+0000","last_peered":"2024-03-21T08:42:31.687899+0000","last_clean":"2024-03-21T08:42:31.687899+0000","last_became_active":"2024-03-21T08:42:29.680101+0000","last_became_peered":"2024-03-21T08:42:29.680101+0000","last_unstale":"2024-03-21T08:42:31.687899+0000","last_undegraded":"2024-03-21T08:42:31.687899+0000","last_fullsized":"2024-03-21T08:42:31.687899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:04:54.658152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046863+0000","last_change":"2024-03-21T08:42:30.684751+0000","last_active":"2024-03-21T08:42:32.046863+0000","last_peered":"2024-03-21T08:42:32.046863+0000","last_clean":"2024-03-21T08:42:32.046863+0000","last_became_active":"2024-03-21T08:42:30.683841+0000","last_became_peered":"2024-03-21T08:42:30.683841+0000","last_unstale":"2024-03-21T08:42:32.046863+0000","last_undegraded":"2024-03-21T08:42:32.046863+0000","last_fullsized":"2024-03-21T08:42:32.046863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:12.042436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046596+0000","last_change":"2024-03-21T08:42:29.689562+0000","last_active":"2024-03-21T08:42:32.046596+0000","last_peered":"2024-03-21T08:42:32.046596+0000","last_clean":"2024-03-21T08:42:32.046596+0000","last_became_active":"2024-03-21T08:42:29.687676+0000","last_became_peered":"2024-03-21T08:42:29.687676+0000","last_unstale":"2024-03-21T08:42:32.046596+0000","last_undegraded":"2024-03-21T08:42:32.046596+0000","last_fullsized":"2024-03-21T08:42:32.046596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:03:39.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750777+0000","last_change":"2024-03-21T08:42:30.682253+0000","last_active":"2024-03-21T08:42:31.750777+0000","last_peered":"2024-03-21T08:42:31.750777+0000","last_clean":"2024-03-21T08:42:31.750777+0000","last_became_active":"2024-03-21T08:42:30.681976+0000","last_became_peered":"2024-03-21T08:42:30.681976+0000","last_unstale":"2024-03-21T08:42:31.750777+0000","last_undegraded":"2024-03-21T08:42:31.750777+0000","last_fullsized":"2024-03-21T08:42:31.750777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:33:36.238601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046694+0000","last_change":"2024-03-21T08:42:29.690366+0000","last_active":"2024-03-21T08:42:32.046694+0000","last_peered":"2024-03-21T08:42:32.046694+0000","last_clean":"2024-03-21T08:42:32.046694+0000","last_became_active":"2024-03-21T08:42:29.690030+0000","last_became_peered":"2024-03-21T08:42:29.690030+0000","last_unstale":"2024-03-21T08:42:32.046694+0000","last_undegraded":"2024-03-21T08:42:32.046694+0000","last_fullsized":"2024-03-21T08:42:32.046694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:57:52.751505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046344+0000","last_change":"2024-03-21T08:42:30.686858+0000","last_active":"2024-03-21T08:42:32.046344+0000","last_peered":"2024-03-21T08:42:32.046344+0000","last_clean":"2024-03-21T08:42:32.046344+0000","last_became_active":"2024-03-21T08:42:30.686587+0000","last_became_peered":"2024-03-21T08:42:30.686587+0000","last_unstale":"2024-03-21T08:42:32.046344+0000","last_undegraded":"2024-03-21T08:42:32.046344+0000","last_fullsized":"2024-03-21T08:42:32.046344+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:11:21.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046657+0000","last_change":"2024-03-21T08:42:29.687057+0000","last_active":"2024-03-21T08:42:32.046657+0000","last_peered":"2024-03-21T08:42:32.046657+0000","last_clean":"2024-03-21T08:42:32.046657+0000","last_became_active":"2024-03-21T08:42:29.686782+0000","last_became_peered":"2024-03-21T08:42:29.686782+0000","last_unstale":"2024-03-21T08:42:32.046657+0000","last_undegraded":"2024-03-21T08:42:32.046657+0000","last_fullsized":"2024-03-21T08:42:32.046657+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:13.487644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687512+0000","last_change":"2024-03-21T08:42:30.695584+0000","last_active":"2024-03-21T08:42:31.687512+0000","last_peered":"2024-03-21T08:42:31.687512+0000","last_clean":"2024-03-21T08:42:31.687512+0000","last_became_active":"2024-03-21T08:42:30.695232+0000","last_became_peered":"2024-03-21T08:42:30.695232+0000","last_unstale":"2024-03-21T08:42:31.687512+0000","last_undegraded":"2024-03-21T08:42:31.687512+0000","last_fullsized":"2024-03-21T08:42:31.687512+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:30.082762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045569+0000","last_change":"2024-03-21T08:42:29.685401+0000","last_active":"2024-03-21T08:42:32.045569+0000","last_peered":"2024-03-21T08:42:32.045569+0000","last_clean":"2024-03-21T08:42:32.045569+0000","last_became_active":"2024-03-21T08:42:29.683391+0000","last_became_peered":"2024-03-21T08:42:29.683391+0000","last_unstale":"2024-03-21T08:42:32.045569+0000","last_undegraded":"2024-03-21T08:42:32.045569+0000","last_fullsized":"2024-03-21T08:42:32.045569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:42.529477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046572+0000","last_change":"2024-03-21T08:42:30.696914+0000","last_active":"2024-03-21T08:42:32.046572+0000","last_peered":"2024-03-21T08:42:32.046572+0000","last_clean":"2024-03-21T08:42:32.046572+0000","last_became_active":"2024-03-21T08:42:30.696632+0000","last_became_peered":"2024-03-21T08:42:30.696632+0000","last_unstale":"2024-03-21T08:42:32.046572+0000","last_undegraded":"2024-03-21T08:42:32.046572+0000","last_fullsized":"2024-03-21T08:42:32.046572+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:58:57.974782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686893+0000","last_change":"2024-03-21T08:42:29.689164+0000","last_active":"2024-03-21T08:42:31.686893+0000","last_peered":"2024-03-21T08:42:31.686893+0000","last_clean":"2024-03-21T08:42:31.686893+0000","last_became_active":"2024-03-21T08:42:29.688812+0000","last_became_peered":"2024-03-21T08:42:29.688812+0000","last_unstale":"2024-03-21T08:42:31.686893+0000","last_undegraded":"2024-03-21T08:42:31.686893+0000","last_fullsized":"2024-03-21T08:42:31.686893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:38.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046583+0000","last_change":"2024-03-21T08:42:30.701754+0000","last_active":"2024-03-21T08:42:32.046583+0000","last_peered":"2024-03-21T08:42:32.046583+0000","last_clean":"2024-03-21T08:42:32.046583+0000","last_became_active":"2024-03-21T08:42:30.701476+0000","last_became_peered":"2024-03-21T08:42:30.701476+0000","last_unstale":"2024-03-21T08:42:32.046583+0000","last_undegraded":"2024-03-21T08:42:32.046583+0000","last_fullsized":"2024-03-21T08:42:32.046583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:32.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687642+0000","last_change":"2024-03-21T08:42:29.683970+0000","last_active":"2024-03-21T08:42:31.687642+0000","last_peered":"2024-03-21T08:42:31.687642+0000","last_clean":"2024-03-21T08:42:31.687642+0000","last_became_active":"2024-03-21T08:42:29.683655+0000","last_became_peered":"2024-03-21T08:42:29.683655+0000","last_unstale":"2024-03-21T08:42:31.687642+0000","last_undegraded":"2024-03-21T08:42:31.687642+0000","last_fullsized":"2024-03-21T08:42:31.687642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:36.488242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687897+0000","last_change":"2024-03-21T08:42:30.693462+0000","last_active":"2024-03-21T08:42:31.687897+0000","last_peered":"2024-03-21T08:42:31.687897+0000","last_clean":"2024-03-21T08:42:31.687897+0000","last_became_active":"2024-03-21T08:42:30.693030+0000","last_became_peered":"2024-03-21T08:42:30.693030+0000","last_unstale":"2024-03-21T08:42:31.687897+0000","last_undegraded":"2024-03-21T08:42:31.687897+0000","last_fullsized":"2024-03-21T08:42:31.687897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:18.960768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046668+0000","last_change":"2024-03-21T08:42:29.683875+0000","last_active":"2024-03-21T08:42:32.046668+0000","last_peered":"2024-03-21T08:42:32.046668+0000","last_clean":"2024-03-21T08:42:32.046668+0000","last_became_active":"2024-03-21T08:42:29.683393+0000","last_became_peered":"2024-03-21T08:42:29.683393+0000","last_unstale":"2024-03-21T08:42:32.046668+0000","last_undegraded":"2024-03-21T08:42:32.046668+0000","last_fullsized":"2024-03-21T08:42:32.046668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:54:19.261503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046630+0000","last_change":"2024-03-21T08:42:30.689908+0000","last_active":"2024-03-21T08:42:32.046630+0000","last_peered":"2024-03-21T08:42:32.046630+0000","last_clean":"2024-03-21T08:42:32.046630+0000","last_became_active":"2024-03-21T08:42:30.689649+0000","last_became_peered":"2024-03-21T08:42:30.689649+0000","last_unstale":"2024-03-21T08:42:32.046630+0000","last_undegraded":"2024-03-21T08:42:32.046630+0000","last_fullsized":"2024-03-21T08:42:32.046630+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:21.214239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686906+0000","last_change":"2024-03-21T08:42:29.685315+0000","last_active":"2024-03-21T08:42:31.686906+0000","last_peered":"2024-03-21T08:42:31.686906+0000","last_clean":"2024-03-21T08:42:31.686906+0000","last_became_active":"2024-03-21T08:42:29.684654+0000","last_became_peered":"2024-03-21T08:42:29.684654+0000","last_unstale":"2024-03-21T08:42:31.686906+0000","last_undegraded":"2024-03-21T08:42:31.686906+0000","last_fullsized":"2024-03-21T08:42:31.686906+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:28:45.351752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687118+0000","last_change":"2024-03-21T08:42:30.680532+0000","last_active":"2024-03-21T08:42:31.687118+0000","last_peered":"2024-03-21T08:42:31.687118+0000","last_clean":"2024-03-21T08:42:31.687118+0000","last_became_active":"2024-03-21T08:42:30.680074+0000","last_became_peered":"2024-03-21T08:42:30.680074+0000","last_unstale":"2024-03-21T08:42:31.687118+0000","last_undegraded":"2024-03-21T08:42:31.687118+0000","last_fullsized":"2024-03-21T08:42:31.687118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:38:25.501198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046948+0000","last_change":"2024-03-21T08:42:29.682271+0000","last_active":"2024-03-21T08:42:32.046948+0000","last_peered":"2024-03-21T08:42:32.046948+0000","last_clean":"2024-03-21T08:42:32.046948+0000","last_became_active":"2024-03-21T08:42:29.681820+0000","last_became_peered":"2024-03-21T08:42:29.681820+0000","last_unstale":"2024-03-21T08:42:32.046948+0000","last_undegraded":"2024-03-21T08:42:32.046948+0000","last_fullsized":"2024-03-21T08:42:32.046948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:54.299526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687052+0000","last_change":"2024-03-21T08:42:30.698905+0000","last_active":"2024-03-21T08:42:31.687052+0000","last_peered":"2024-03-21T08:42:31.687052+0000","last_clean":"2024-03-21T08:42:31.687052+0000","last_became_active":"2024-03-21T08:42:30.698438+0000","last_became_peered":"2024-03-21T08:42:30.698438+0000","last_unstale":"2024-03-21T08:42:31.687052+0000","last_undegraded":"2024-03-21T08:42:31.687052+0000","last_fullsized":"2024-03-21T08:42:31.687052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:04:31.289373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686603+0000","last_change":"2024-03-21T08:42:29.681387+0000","last_active":"2024-03-21T08:42:31.686603+0000","last_peered":"2024-03-21T08:42:31.686603+0000","last_clean":"2024-03-21T08:42:31.686603+0000","last_became_active":"2024-03-21T08:42:29.681045+0000","last_became_peered":"2024-03-21T08:42:29.681045+0000","last_unstale":"2024-03-21T08:42:31.686603+0000","last_undegraded":"2024-03-21T08:42:31.686603+0000","last_fullsized":"2024-03-21T08:42:31.686603+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:54.348736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750787+0000","last_change":"2024-03-21T08:42:30.696906+0000","last_active":"2024-03-21T08:42:31.750787+0000","last_peered":"2024-03-21T08:42:31.750787+0000","last_clean":"2024-03-21T08:42:31.750787+0000","last_became_active":"2024-03-21T08:42:30.696649+0000","last_became_peered":"2024-03-21T08:42:30.696649+0000","last_unstale":"2024-03-21T08:42:31.750787+0000","last_undegraded":"2024-03-21T08:42:31.750787+0000","last_fullsized":"2024-03-21T08:42:31.750787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:05.195688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686426+0000","last_change":"2024-03-21T08:42:29.686227+0000","last_active":"2024-03-21T08:42:31.686426+0000","last_peered":"2024-03-21T08:42:31.686426+0000","last_clean":"2024-03-21T08:42:31.686426+0000","last_became_active":"2024-03-21T08:42:29.685952+0000","last_became_peered":"2024-03-21T08:42:29.685952+0000","last_unstale":"2024-03-21T08:42:31.686426+0000","last_undegraded":"2024-03-21T08:42:31.686426+0000","last_fullsized":"2024-03-21T08:42:31.686426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:27.633396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688555+0000","last_change":"2024-03-21T08:42:30.687101+0000","last_active":"2024-03-21T08:42:31.688555+0000","last_peered":"2024-03-21T08:42:31.688555+0000","last_clean":"2024-03-21T08:42:31.688555+0000","last_became_active":"2024-03-21T08:42:30.686821+0000","last_became_peered":"2024-03-21T08:42:30.686821+0000","last_unstale":"2024-03-21T08:42:31.688555+0000","last_undegraded":"2024-03-21T08:42:31.688555+0000","last_fullsized":"2024-03-21T08:42:31.688555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:03:38.474542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688727+0000","last_change":"2024-03-21T08:42:29.689729+0000","last_active":"2024-03-21T08:42:31.688727+0000","last_peered":"2024-03-21T08:42:31.688727+0000","last_clean":"2024-03-21T08:42:31.688727+0000","last_became_active":"2024-03-21T08:42:29.689417+0000","last_became_peered":"2024-03-21T08:42:29.689417+0000","last_unstale":"2024-03-21T08:42:31.688727+0000","last_undegraded":"2024-03-21T08:42:31.688727+0000","last_fullsized":"2024-03-21T08:42:31.688727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:16:03.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046786+0000","last_change":"2024-03-21T08:42:30.696977+0000","last_active":"2024-03-21T08:42:32.046786+0000","last_peered":"2024-03-21T08:42:32.046786+0000","last_clean":"2024-03-21T08:42:32.046786+0000","last_became_active":"2024-03-21T08:42:30.696683+0000","last_became_peered":"2024-03-21T08:42:30.696683+0000","last_unstale":"2024-03-21T08:42:32.046786+0000","last_undegraded":"2024-03-21T08:42:32.046786+0000","last_fullsized":"2024-03-21T08:42:32.046786+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686313+0000","last_change":"2024-03-21T08:42:29.681441+0000","last_active":"2024-03-21T08:42:31.686313+0000","last_peered":"2024-03-21T08:42:31.686313+0000","last_clean":"2024-03-21T08:42:31.686313+0000","last_became_active":"2024-03-21T08:42:29.680958+0000","last_became_peered":"2024-03-21T08:42:29.680958+0000","last_unstale":"2024-03-21T08:42:31.686313+0000","last_undegraded":"2024-03-21T08:42:31.686313+0000","last_fullsized":"2024-03-21T08:42:31.686313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:17.224372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046923+0000","last_change":"2024-03-21T08:42:30.690979+0000","last_active":"2024-03-21T08:42:32.046923+0000","last_peered":"2024-03-21T08:42:32.046923+0000","last_clean":"2024-03-21T08:42:32.046923+0000","last_became_active":"2024-03-21T08:42:30.690659+0000","last_became_peered":"2024-03-21T08:42:30.690659+0000","last_unstale":"2024-03-21T08:42:32.046923+0000","last_undegraded":"2024-03-21T08:42:32.046923+0000","last_fullsized":"2024-03-21T08:42:32.046923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:24:58.246991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687767+0000","last_change":"2024-03-21T08:42:29.686849+0000","last_active":"2024-03-21T08:42:31.687767+0000","last_peered":"2024-03-21T08:42:31.687767+0000","last_clean":"2024-03-21T08:42:31.687767+0000","last_became_active":"2024-03-21T08:42:29.686581+0000","last_became_peered":"2024-03-21T08:42:29.686581+0000","last_unstale":"2024-03-21T08:42:31.687767+0000","last_undegraded":"2024-03-21T08:42:31.687767+0000","last_fullsized":"2024-03-21T08:42:31.687767+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:01.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688218+0000","last_change":"2024-03-21T08:42:30.699470+0000","last_active":"2024-03-21T08:42:31.688218+0000","last_peered":"2024-03-21T08:42:31.688218+0000","last_clean":"2024-03-21T08:42:31.688218+0000","last_became_active":"2024-03-21T08:42:30.699169+0000","last_became_peered":"2024-03-21T08:42:30.699169+0000","last_unstale":"2024-03-21T08:42:31.688218+0000","last_undegraded":"2024-03-21T08:42:31.688218+0000","last_fullsized":"2024-03-21T08:42:31.688218+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:36.190026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687019+0000","last_change":"2024-03-21T08:42:29.680363+0000","last_active":"2024-03-21T08:42:31.687019+0000","last_peered":"2024-03-21T08:42:31.687019+0000","last_clean":"2024-03-21T08:42:31.687019+0000","last_became_active":"2024-03-21T08:42:29.680096+0000","last_became_peered":"2024-03-21T08:42:29.680096+0000","last_unstale":"2024-03-21T08:42:31.687019+0000","last_undegraded":"2024-03-21T08:42:31.687019+0000","last_fullsized":"2024-03-21T08:42:31.687019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:30:03.838556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046835+0000","last_change":"2024-03-21T08:42:30.685856+0000","last_active":"2024-03-21T08:42:32.046835+0000","last_peered":"2024-03-21T08:42:32.046835+0000","last_clean":"2024-03-21T08:42:32.046835+0000","last_became_active":"2024-03-21T08:42:30.685578+0000","last_became_peered":"2024-03-21T08:42:30.685578+0000","last_unstale":"2024-03-21T08:42:32.046835+0000","last_undegraded":"2024-03-21T08:42:32.046835+0000","last_fullsized":"2024-03-21T08:42:32.046835+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:18.960768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750952+0000","last_change":"2024-03-21T08:42:29.689582+0000","last_active":"2024-03-21T08:42:31.750952+0000","last_peered":"2024-03-21T08:42:31.750952+0000","last_clean":"2024-03-21T08:42:31.750952+0000","last_became_active":"2024-03-21T08:42:29.689346+0000","last_became_peered":"2024-03-21T08:42:29.689346+0000","last_unstale":"2024-03-21T08:42:31.750952+0000","last_undegraded":"2024-03-21T08:42:31.750952+0000","last_fullsized":"2024-03-21T08:42:31.750952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:00:31.687764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751089+0000","last_change":"2024-03-21T08:42:30.691466+0000","last_active":"2024-03-21T08:42:31.751089+0000","last_peered":"2024-03-21T08:42:31.751089+0000","last_clean":"2024-03-21T08:42:31.751089+0000","last_became_active":"2024-03-21T08:42:30.688723+0000","last_became_peered":"2024-03-21T08:42:30.688723+0000","last_unstale":"2024-03-21T08:42:31.751089+0000","last_undegraded":"2024-03-21T08:42:31.751089+0000","last_fullsized":"2024-03-21T08:42:31.751089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:51:06.301613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750901+0000","last_change":"2024-03-21T08:42:29.685783+0000","last_active":"2024-03-21T08:42:31.750901+0000","last_peered":"2024-03-21T08:42:31.750901+0000","last_clean":"2024-03-21T08:42:31.750901+0000","last_became_active":"2024-03-21T08:42:29.685512+0000","last_became_peered":"2024-03-21T08:42:29.685512+0000","last_unstale":"2024-03-21T08:42:31.750901+0000","last_undegraded":"2024-03-21T08:42:31.750901+0000","last_fullsized":"2024-03-21T08:42:31.750901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:40.758395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046075+0000","last_change":"2024-03-21T08:42:30.697765+0000","last_active":"2024-03-21T08:42:32.046075+0000","last_peered":"2024-03-21T08:42:32.046075+0000","last_clean":"2024-03-21T08:42:32.046075+0000","last_became_active":"2024-03-21T08:42:30.695427+0000","last_became_peered":"2024-03-21T08:42:30.695427+0000","last_unstale":"2024-03-21T08:42:32.046075+0000","last_undegraded":"2024-03-21T08:42:32.046075+0000","last_fullsized":"2024-03-21T08:42:32.046075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:16:08.486846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045885+0000","last_change":"2024-03-21T08:42:29.681258+0000","last_active":"2024-03-21T08:42:32.045885+0000","last_peered":"2024-03-21T08:42:32.045885+0000","last_clean":"2024-03-21T08:42:32.045885+0000","last_became_active":"2024-03-21T08:42:29.680977+0000","last_became_peered":"2024-03-21T08:42:29.680977+0000","last_unstale":"2024-03-21T08:42:32.045885+0000","last_undegraded":"2024-03-21T08:42:32.045885+0000","last_fullsized":"2024-03-21T08:42:32.045885+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046550+0000","last_change":"2024-03-21T08:42:30.679950+0000","last_active":"2024-03-21T08:42:32.046550+0000","last_peered":"2024-03-21T08:42:32.046550+0000","last_clean":"2024-03-21T08:42:32.046550+0000","last_became_active":"2024-03-21T08:42:30.679672+0000","last_became_peered":"2024-03-21T08:42:30.679672+0000","last_unstale":"2024-03-21T08:42:32.046550+0000","last_undegraded":"2024-03-21T08:42:32.046550+0000","last_fullsized":"2024-03-21T08:42:32.046550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:05.195688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045674+0000","last_change":"2024-03-21T08:42:29.687355+0000","last_active":"2024-03-21T08:42:32.045674+0000","last_peered":"2024-03-21T08:42:32.045674+0000","last_clean":"2024-03-21T08:42:32.045674+0000","last_became_active":"2024-03-21T08:42:29.687072+0000","last_became_peered":"2024-03-21T08:42:29.687072+0000","last_unstale":"2024-03-21T08:42:32.045674+0000","last_undegraded":"2024-03-21T08:42:32.045674+0000","last_fullsized":"2024-03-21T08:42:32.045674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:40.758395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688114+0000","last_change":"2024-03-21T08:42:30.699318+0000","last_active":"2024-03-21T08:42:31.688114+0000","last_peered":"2024-03-21T08:42:31.688114+0000","last_clean":"2024-03-21T08:42:31.688114+0000","last_became_active":"2024-03-21T08:42:30.699015+0000","last_became_peered":"2024-03-21T08:42:30.699015+0000","last_unstale":"2024-03-21T08:42:31.688114+0000","last_undegraded":"2024-03-21T08:42:31.688114+0000","last_fullsized":"2024-03-21T08:42:31.688114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:57:55.777767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046238+0000","last_change":"2024-03-21T08:42:29.685473+0000","last_active":"2024-03-21T08:42:32.046238+0000","last_peered":"2024-03-21T08:42:32.046238+0000","last_clean":"2024-03-21T08:42:32.046238+0000","last_became_active":"2024-03-21T08:42:29.685208+0000","last_became_peered":"2024-03-21T08:42:29.685208+0000","last_unstale":"2024-03-21T08:42:32.046238+0000","last_undegraded":"2024-03-21T08:42:32.046238+0000","last_fullsized":"2024-03-21T08:42:32.046238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:45.341543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687622+0000","last_change":"2024-03-21T08:42:30.681206+0000","last_active":"2024-03-21T08:42:31.687622+0000","last_peered":"2024-03-21T08:42:31.687622+0000","last_clean":"2024-03-21T08:42:31.687622+0000","last_became_active":"2024-03-21T08:42:30.680936+0000","last_became_peered":"2024-03-21T08:42:30.680936+0000","last_unstale":"2024-03-21T08:42:31.687622+0000","last_undegraded":"2024-03-21T08:42:31.687622+0000","last_fullsized":"2024-03-21T08:42:31.687622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:28:41.046722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688329+0000","last_change":"2024-03-21T08:42:29.692572+0000","last_active":"2024-03-21T08:42:31.688329+0000","last_peered":"2024-03-21T08:42:31.688329+0000","last_clean":"2024-03-21T08:42:31.688329+0000","last_became_active":"2024-03-21T08:42:29.692230+0000","last_became_peered":"2024-03-21T08:42:29.692230+0000","last_unstale":"2024-03-21T08:42:31.688329+0000","last_undegraded":"2024-03-21T08:42:31.688329+0000","last_fullsized":"2024-03-21T08:42:31.688329+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:03.893918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687310+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:42:31.687310+0000","last_peered":"2024-03-21T08:42:31.687310+0000","last_clean":"2024-03-21T08:42:31.687310+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:42:31.687310+0000","last_undegraded":"2024-03-21T08:42:31.687310+0000","last_fullsized":"2024-03-21T08:42:31.687310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1052672,"data_stored":784071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":192603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509463,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":192091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":192111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":193667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542167,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1040384,"data_stored":782423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:42:37.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:42:37.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:42:39.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:42:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 Processor -- start 2024-03-21T08:42:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 -- start start 2024-03-21T08:42:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ec00af060 0x7f0ec00cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec015c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec015bf00 con 0x7f0ec00b34f0 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00ce6e0 con 0x7f0ec00af060 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ec5400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00b3980 con 0x7f0ec00b5520 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ec00af060 0x7f0ec00cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ec00af060 0x7f0ec00cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43214/0 (socket says 172.21.15.28:43214) 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 -- 172.21.15.28:0/2041040670 learned_addr learned my addr 172.21.15.28:0/2041040670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebf200640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec015c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 -- 172.21.15.28:0/2041040670 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 msgr2=0x7f0ec015c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec015c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 -- 172.21.15.28:0/2041040670 >> v1:172.21.15.151:6789/0 conn(0x7f0ec00b34f0 legacy=0x7f0ec00b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.571+0000 7f0ebe800640 1 -- 172.21.15.28:0/2041040670 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00cdad0 con 0x7f0ec00af060 2024-03-21T08:42:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ebe800640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00af060 0x7f0ec00cd380 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009e50 tx=0x7f0eb001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b12894eafdbae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ebde00640 1 -- 172.21.15.28:0/2041040670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00a5660 con 0x7f0ec00af060 2024-03-21T08:42:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ebde00640 1 -- 172.21.15.28:0/2041040670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb00a5800 con 0x7f0ec00af060 2024-03-21T08:42:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ebde00640 1 -- 172.21.15.28:0/2041040670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00a5bd0 con 0x7f0ec00af060 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 -- 172.21.15.28:0/2041040670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00af060 msgr2=0x7f0ec00cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00af060 0x7f0ec00cd380 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009e50 tx=0x7f0eb001c9a0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 -- 172.21.15.28:0/2041040670 shutdown_connections 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec015c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 --2- 172.21.15.28:0/2041040670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00af060 0x7f0ec00cd380 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 -- 172.21.15.28:0/2041040670 >> 172.21.15.28:0/2041040670 conn(0x7f0ec00cc230 msgr2=0x7f0ec00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 -- 172.21.15.28:0/2041040670 shutdown_connections 2024-03-21T08:42:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.575+0000 7f0ec5400640 1 -- 172.21.15.28:0/2041040670 wait complete. 2024-03-21T08:42:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 Processor -- start 2024-03-21T08:42:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- start start 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec013a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ec01503f0 0x7f0ec0155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015d0d0 con 0x7f0ec00b34f0 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015cf50 con 0x7f0ec01503f0 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec015d250 con 0x7f0ec00b5520 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43230/0 (socket says 172.21.15.28:43230) 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 -- 172.21.15.28:0/3199840787 learned_addr learned my addr 172.21.15.28:0/3199840787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0eb5e00640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec013a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebf200640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ec01503f0 0x7f0ec0155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 -- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 msgr2=0x7f0ec013a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec013a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 -- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ec01503f0 msgr2=0x7f0ec0155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ec01503f0 0x7f0ec0155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 -- 172.21.15.28:0/3199840787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec0150b60 con 0x7f0ec00b34f0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ebe800640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0eb001a5d0 tx=0x7f0eb0019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00bc020 con 0x7f0ec00b34f0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb00b8670 con 0x7f0ec00b34f0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec0150e40 con 0x7f0ec00b34f0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00b8a60 con 0x7f0ec00b34f0 2024-03-21T08:42:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.579+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec0151400 con 0x7f0ec00b34f0 2024-03-21T08:42:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.583+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec004f410 con 0x7f0ec00b34f0 2024-03-21T08:42:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.583+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0eb00bf020 con 0x7f0ec00b34f0 2024-03-21T08:42:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.583+0000 7f0eb7400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 0x7f0e9c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.583+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0eb0120c20 con 0x7f0ec00b34f0 2024-03-21T08:42:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.583+0000 7f0eb5e00640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 0x7f0e9c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.587+0000 7f0eb5e00640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 0x7f0e9c06a010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0153b20 tx=0x7f0ea8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.587+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0eb0016620 con 0x7f0ec00b34f0 2024-03-21T08:42:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.751+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ec00b28f0 con 0x7f0e9c067b50 2024-03-21T08:42:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.759+0000 7f0eb7400640 1 -- 172.21.15.28:0/3199840787 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251910 (secure 0 0 0) 0x7f0ec00b28f0 con 0x7f0e9c067b50 2024-03-21T08:42:39.763 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:42:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 msgr2=0x7f0e9c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 0x7f0e9c06a010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0153b20 tx=0x7f0ea8009040 comp rx=0 tx=0).stop 2024-03-21T08:42:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 msgr2=0x7f0ec014a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0eb001a5d0 tx=0x7f0eb0019740 comp rx=0 tx=0).stop 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 shutdown_connections 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0ec01503f0 0x7f0ec0155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e9c067b50 0x7f0e9c06a010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0ec00b5520 0x7f0ec013a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3199840787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec00b34f0 0x7f0ec014a650 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 >> 172.21.15.28:0/3199840787 conn(0x7f0ec00cc230 msgr2=0x7f0ec015eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 shutdown_connections 2024-03-21T08:42:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.763+0000 7f0ec5400640 1 -- 172.21.15.28:0/3199840787 wait complete. 2024-03-21T08:42:39.785 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-03-21T08:42:39.738140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237516,"kb_used_data":4680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737204,"statfs":{"total":773094113280,"available":772850896896,"internally_reserved":0,"allocated":4792320,"data_stored":2721724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.070778"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046862+0000","last_change":"2024-03-21T08:42:30.695871+0000","last_active":"2024-03-21T08:42:32.046862+0000","last_peered":"2024-03-21T08:42:32.046862+0000","last_clean":"2024-03-21T08:42:32.046862+0000","last_became_active":"2024-03-21T08:42:30.695549+0000","last_became_peered":"2024-03-21T08:42:30.695549+0000","last_unstale":"2024-03-21T08:42:32.046862+0000","last_undegraded":"2024-03-21T08:42:32.046862+0000","last_fullsized":"2024-03-21T08:42:32.046862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:50:31.978070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046492+0000","last_change":"2024-03-21T08:42:30.687207+0000","last_active":"2024-03-21T08:42:32.046492+0000","last_peered":"2024-03-21T08:42:32.046492+0000","last_clean":"2024-03-21T08:42:32.046492+0000","last_became_active":"2024-03-21T08:42:30.686933+0000","last_became_peered":"2024-03-21T08:42:30.686933+0000","last_unstale":"2024-03-21T08:42:32.046492+0000","last_undegraded":"2024-03-21T08:42:32.046492+0000","last_fullsized":"2024-03-21T08:42:32.046492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:40.800907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751168+0000","last_change":"2024-03-21T08:42:30.682877+0000","last_active":"2024-03-21T08:42:31.751168+0000","last_peered":"2024-03-21T08:42:31.751168+0000","last_clean":"2024-03-21T08:42:31.751168+0000","last_became_active":"2024-03-21T08:42:30.682612+0000","last_became_peered":"2024-03-21T08:42:30.682612+0000","last_unstale":"2024-03-21T08:42:31.751168+0000","last_undegraded":"2024-03-21T08:42:31.751168+0000","last_fullsized":"2024-03-21T08:42:31.751168+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:12.042436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686840+0000","last_change":"2024-03-21T08:42:30.678176+0000","last_active":"2024-03-21T08:42:31.686840+0000","last_peered":"2024-03-21T08:42:31.686840+0000","last_clean":"2024-03-21T08:42:31.686840+0000","last_became_active":"2024-03-21T08:42:30.677904+0000","last_became_peered":"2024-03-21T08:42:30.677904+0000","last_unstale":"2024-03-21T08:42:31.686840+0000","last_undegraded":"2024-03-21T08:42:31.686840+0000","last_fullsized":"2024-03-21T08:42:31.686840+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:55.306648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686403+0000","last_change":"2024-03-21T08:42:30.677271+0000","last_active":"2024-03-21T08:42:31.686403+0000","last_peered":"2024-03-21T08:42:31.686403+0000","last_clean":"2024-03-21T08:42:31.686403+0000","last_became_active":"2024-03-21T08:42:30.676973+0000","last_became_peered":"2024-03-21T08:42:30.676973+0000","last_unstale":"2024-03-21T08:42:31.686403+0000","last_undegraded":"2024-03-21T08:42:31.686403+0000","last_fullsized":"2024-03-21T08:42:31.686403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:39:57.865608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046416+0000","last_change":"2024-03-21T08:42:30.686840+0000","last_active":"2024-03-21T08:42:32.046416+0000","last_peered":"2024-03-21T08:42:32.046416+0000","last_clean":"2024-03-21T08:42:32.046416+0000","last_became_active":"2024-03-21T08:42:30.686561+0000","last_became_peered":"2024-03-21T08:42:30.686561+0000","last_unstale":"2024-03-21T08:42:32.046416+0000","last_undegraded":"2024-03-21T08:42:32.046416+0000","last_fullsized":"2024-03-21T08:42:32.046416+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:55:42.547751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687096+0000","last_change":"2024-03-21T08:42:30.678888+0000","last_active":"2024-03-21T08:42:31.687096+0000","last_peered":"2024-03-21T08:42:31.687096+0000","last_clean":"2024-03-21T08:42:31.687096+0000","last_became_active":"2024-03-21T08:42:30.678604+0000","last_became_peered":"2024-03-21T08:42:30.678604+0000","last_unstale":"2024-03-21T08:42:31.687096+0000","last_undegraded":"2024-03-21T08:42:31.687096+0000","last_fullsized":"2024-03-21T08:42:31.687096+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:11:21.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046263+0000","last_change":"2024-03-21T08:42:30.691873+0000","last_active":"2024-03-21T08:42:32.046263+0000","last_peered":"2024-03-21T08:42:32.046263+0000","last_clean":"2024-03-21T08:42:32.046263+0000","last_became_active":"2024-03-21T08:42:30.687997+0000","last_became_peered":"2024-03-21T08:42:30.687997+0000","last_unstale":"2024-03-21T08:42:32.046263+0000","last_undegraded":"2024-03-21T08:42:32.046263+0000","last_fullsized":"2024-03-21T08:42:32.046263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:24.326977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751237+0000","last_change":"2024-03-21T08:42:30.685358+0000","last_active":"2024-03-21T08:42:31.751237+0000","last_peered":"2024-03-21T08:42:31.751237+0000","last_clean":"2024-03-21T08:42:31.751237+0000","last_became_active":"2024-03-21T08:42:30.685071+0000","last_became_peered":"2024-03-21T08:42:30.685071+0000","last_unstale":"2024-03-21T08:42:31.751237+0000","last_undegraded":"2024-03-21T08:42:31.751237+0000","last_fullsized":"2024-03-21T08:42:31.751237+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:04:04.312171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687195+0000","last_change":"2024-03-21T08:42:30.695493+0000","last_active":"2024-03-21T08:42:31.687195+0000","last_peered":"2024-03-21T08:42:31.687195+0000","last_clean":"2024-03-21T08:42:31.687195+0000","last_became_active":"2024-03-21T08:42:30.695195+0000","last_became_peered":"2024-03-21T08:42:30.695195+0000","last_unstale":"2024-03-21T08:42:31.687195+0000","last_undegraded":"2024-03-21T08:42:31.687195+0000","last_fullsized":"2024-03-21T08:42:31.687195+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:58:57.974782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687736+0000","last_change":"2024-03-21T08:42:30.688105+0000","last_active":"2024-03-21T08:42:31.687736+0000","last_peered":"2024-03-21T08:42:31.687736+0000","last_clean":"2024-03-21T08:42:31.687736+0000","last_became_active":"2024-03-21T08:42:30.686104+0000","last_became_peered":"2024-03-21T08:42:30.686104+0000","last_unstale":"2024-03-21T08:42:31.687736+0000","last_undegraded":"2024-03-21T08:42:31.687736+0000","last_fullsized":"2024-03-21T08:42:31.687736+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:38:41.370318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750802+0000","last_change":"2024-03-21T08:42:30.689408+0000","last_active":"2024-03-21T08:42:31.750802+0000","last_peered":"2024-03-21T08:42:31.750802+0000","last_clean":"2024-03-21T08:42:31.750802+0000","last_became_active":"2024-03-21T08:42:30.689100+0000","last_became_peered":"2024-03-21T08:42:30.689100+0000","last_unstale":"2024-03-21T08:42:31.750802+0000","last_undegraded":"2024-03-21T08:42:31.750802+0000","last_fullsized":"2024-03-21T08:42:31.750802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:21.214239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687225+0000","last_change":"2024-03-21T08:42:30.698030+0000","last_active":"2024-03-21T08:42:31.687225+0000","last_peered":"2024-03-21T08:42:31.687225+0000","last_clean":"2024-03-21T08:42:31.687225+0000","last_became_active":"2024-03-21T08:42:30.697796+0000","last_became_peered":"2024-03-21T08:42:30.697796+0000","last_unstale":"2024-03-21T08:42:31.687225+0000","last_undegraded":"2024-03-21T08:42:31.687225+0000","last_fullsized":"2024-03-21T08:42:31.687225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:32.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046166+0000","last_change":"2024-03-21T08:42:30.687179+0000","last_active":"2024-03-21T08:42:32.046166+0000","last_peered":"2024-03-21T08:42:32.046166+0000","last_clean":"2024-03-21T08:42:32.046166+0000","last_became_active":"2024-03-21T08:42:30.686870+0000","last_became_peered":"2024-03-21T08:42:30.686870+0000","last_unstale":"2024-03-21T08:42:32.046166+0000","last_undegraded":"2024-03-21T08:42:32.046166+0000","last_fullsized":"2024-03-21T08:42:32.046166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:03:40.989778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686963+0000","last_change":"2024-03-21T08:42:30.686337+0000","last_active":"2024-03-21T08:42:31.686963+0000","last_peered":"2024-03-21T08:42:31.686963+0000","last_clean":"2024-03-21T08:42:31.686963+0000","last_became_active":"2024-03-21T08:42:30.686026+0000","last_became_peered":"2024-03-21T08:42:30.686026+0000","last_unstale":"2024-03-21T08:42:31.686963+0000","last_undegraded":"2024-03-21T08:42:31.686963+0000","last_fullsized":"2024-03-21T08:42:31.686963+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:22:24.326977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686779+0000","last_change":"2024-03-21T08:42:30.695084+0000","last_active":"2024-03-21T08:42:31.686779+0000","last_peered":"2024-03-21T08:42:31.686779+0000","last_clean":"2024-03-21T08:42:31.686779+0000","last_became_active":"2024-03-21T08:42:30.694799+0000","last_became_peered":"2024-03-21T08:42:30.694799+0000","last_unstale":"2024-03-21T08:42:31.686779+0000","last_undegraded":"2024-03-21T08:42:31.686779+0000","last_fullsized":"2024-03-21T08:42:31.686779+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:29:30.921384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687929+0000","last_change":"2024-03-21T08:42:30.700752+0000","last_active":"2024-03-21T08:42:31.687929+0000","last_peered":"2024-03-21T08:42:31.687929+0000","last_clean":"2024-03-21T08:42:31.687929+0000","last_became_active":"2024-03-21T08:42:30.698996+0000","last_became_peered":"2024-03-21T08:42:30.698996+0000","last_unstale":"2024-03-21T08:42:31.687929+0000","last_undegraded":"2024-03-21T08:42:31.687929+0000","last_fullsized":"2024-03-21T08:42:31.687929+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:16:19.492004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046260+0000","last_change":"2024-03-21T08:42:30.694366+0000","last_active":"2024-03-21T08:42:32.046260+0000","last_peered":"2024-03-21T08:42:32.046260+0000","last_clean":"2024-03-21T08:42:32.046260+0000","last_became_active":"2024-03-21T08:42:30.693997+0000","last_became_peered":"2024-03-21T08:42:30.693997+0000","last_unstale":"2024-03-21T08:42:32.046260+0000","last_undegraded":"2024-03-21T08:42:32.046260+0000","last_fullsized":"2024-03-21T08:42:32.046260+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:06:29.004037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686922+0000","last_change":"2024-03-21T08:42:30.691546+0000","last_active":"2024-03-21T08:42:31.686922+0000","last_peered":"2024-03-21T08:42:31.686922+0000","last_clean":"2024-03-21T08:42:31.686922+0000","last_became_active":"2024-03-21T08:42:30.691237+0000","last_became_peered":"2024-03-21T08:42:30.691237+0000","last_unstale":"2024-03-21T08:42:31.686922+0000","last_undegraded":"2024-03-21T08:42:31.686922+0000","last_fullsized":"2024-03-21T08:42:31.686922+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:04.901041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046219+0000","last_change":"2024-03-21T08:42:30.698276+0000","last_active":"2024-03-21T08:42:32.046219+0000","last_peered":"2024-03-21T08:42:32.046219+0000","last_clean":"2024-03-21T08:42:32.046219+0000","last_became_active":"2024-03-21T08:42:30.695309+0000","last_became_peered":"2024-03-21T08:42:30.695309+0000","last_unstale":"2024-03-21T08:42:32.046219+0000","last_undegraded":"2024-03-21T08:42:32.046219+0000","last_fullsized":"2024-03-21T08:42:32.046219+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:03.053281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687785+0000","last_change":"2024-03-21T08:42:30.694593+0000","last_active":"2024-03-21T08:42:31.687785+0000","last_peered":"2024-03-21T08:42:31.687785+0000","last_clean":"2024-03-21T08:42:31.687785+0000","last_became_active":"2024-03-21T08:42:30.694289+0000","last_became_peered":"2024-03-21T08:42:30.694289+0000","last_unstale":"2024-03-21T08:42:31.687785+0000","last_undegraded":"2024-03-21T08:42:31.687785+0000","last_fullsized":"2024-03-21T08:42:31.687785+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046761+0000","last_change":"2024-03-21T08:42:30.687038+0000","last_active":"2024-03-21T08:42:32.046761+0000","last_peered":"2024-03-21T08:42:32.046761+0000","last_clean":"2024-03-21T08:42:32.046761+0000","last_became_active":"2024-03-21T08:42:30.686689+0000","last_became_peered":"2024-03-21T08:42:30.686689+0000","last_unstale":"2024-03-21T08:42:32.046761+0000","last_undegraded":"2024-03-21T08:42:32.046761+0000","last_fullsized":"2024-03-21T08:42:32.046761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:51:06.301613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688618+0000","last_change":"2024-03-21T08:42:30.696890+0000","last_active":"2024-03-21T08:42:31.688618+0000","last_peered":"2024-03-21T08:42:31.688618+0000","last_clean":"2024-03-21T08:42:31.688618+0000","last_became_active":"2024-03-21T08:42:30.696103+0000","last_became_peered":"2024-03-21T08:42:30.696103+0000","last_unstale":"2024-03-21T08:42:31.688618+0000","last_undegraded":"2024-03-21T08:42:31.688618+0000","last_fullsized":"2024-03-21T08:42:31.688618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:51:38.815895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686521+0000","last_change":"2024-03-21T08:42:30.695584+0000","last_active":"2024-03-21T08:42:31.686521+0000","last_peered":"2024-03-21T08:42:31.686521+0000","last_clean":"2024-03-21T08:42:31.686521+0000","last_became_active":"2024-03-21T08:42:30.695327+0000","last_became_peered":"2024-03-21T08:42:30.695327+0000","last_unstale":"2024-03-21T08:42:31.686521+0000","last_undegraded":"2024-03-21T08:42:31.686521+0000","last_fullsized":"2024-03-21T08:42:31.686521+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:14.494767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046526+0000","last_change":"2024-03-21T08:42:30.694013+0000","last_active":"2024-03-21T08:42:32.046526+0000","last_peered":"2024-03-21T08:42:32.046526+0000","last_clean":"2024-03-21T08:42:32.046526+0000","last_became_active":"2024-03-21T08:42:30.693709+0000","last_became_peered":"2024-03-21T08:42:30.693709+0000","last_unstale":"2024-03-21T08:42:32.046526+0000","last_undegraded":"2024-03-21T08:42:32.046526+0000","last_fullsized":"2024-03-21T08:42:32.046526+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:22:22.827282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047041+0000","last_change":"2024-03-21T08:42:30.693187+0000","last_active":"2024-03-21T08:42:32.047041+0000","last_peered":"2024-03-21T08:42:32.047041+0000","last_clean":"2024-03-21T08:42:32.047041+0000","last_became_active":"2024-03-21T08:42:30.690678+0000","last_became_peered":"2024-03-21T08:42:30.690678+0000","last_unstale":"2024-03-21T08:42:32.047041+0000","last_undegraded":"2024-03-21T08:42:32.047041+0000","last_fullsized":"2024-03-21T08:42:32.047041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:03:29.930403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046144+0000","last_change":"2024-03-21T08:42:30.694219+0000","last_active":"2024-03-21T08:42:32.046144+0000","last_peered":"2024-03-21T08:42:32.046144+0000","last_clean":"2024-03-21T08:42:32.046144+0000","last_became_active":"2024-03-21T08:42:30.688701+0000","last_became_peered":"2024-03-21T08:42:30.688701+0000","last_unstale":"2024-03-21T08:42:32.046144+0000","last_undegraded":"2024-03-21T08:42:32.046144+0000","last_fullsized":"2024-03-21T08:42:32.046144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:30:56.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688494+0000","last_change":"2024-03-21T08:42:30.690686+0000","last_active":"2024-03-21T08:42:31.688494+0000","last_peered":"2024-03-21T08:42:31.688494+0000","last_clean":"2024-03-21T08:42:31.688494+0000","last_became_active":"2024-03-21T08:42:30.690472+0000","last_became_peered":"2024-03-21T08:42:30.690472+0000","last_unstale":"2024-03-21T08:42:31.688494+0000","last_undegraded":"2024-03-21T08:42:31.688494+0000","last_fullsized":"2024-03-21T08:42:31.688494+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:50:31.978070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046068+0000","last_change":"2024-03-21T08:42:30.693396+0000","last_active":"2024-03-21T08:42:32.046068+0000","last_peered":"2024-03-21T08:42:32.046068+0000","last_clean":"2024-03-21T08:42:32.046068+0000","last_became_active":"2024-03-21T08:42:30.693136+0000","last_became_peered":"2024-03-21T08:42:30.693136+0000","last_unstale":"2024-03-21T08:42:32.046068+0000","last_undegraded":"2024-03-21T08:42:32.046068+0000","last_fullsized":"2024-03-21T08:42:32.046068+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:28.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045777+0000","last_change":"2024-03-21T08:42:30.691385+0000","last_active":"2024-03-21T08:42:32.045777+0000","last_peered":"2024-03-21T08:42:32.045777+0000","last_clean":"2024-03-21T08:42:32.045777+0000","last_became_active":"2024-03-21T08:42:30.690817+0000","last_became_peered":"2024-03-21T08:42:30.690817+0000","last_unstale":"2024-03-21T08:42:32.045777+0000","last_undegraded":"2024-03-21T08:42:32.045777+0000","last_fullsized":"2024-03-21T08:42:32.045777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:16:04.376219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688690+0000","last_change":"2024-03-21T08:42:30.696929+0000","last_active":"2024-03-21T08:42:31.688690+0000","last_peered":"2024-03-21T08:42:31.688690+0000","last_clean":"2024-03-21T08:42:31.688690+0000","last_became_active":"2024-03-21T08:42:30.696470+0000","last_became_peered":"2024-03-21T08:42:30.696470+0000","last_unstale":"2024-03-21T08:42:31.688690+0000","last_undegraded":"2024-03-21T08:42:31.688690+0000","last_fullsized":"2024-03-21T08:42:31.688690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:41:08.879588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046063+0000","last_change":"2024-03-21T08:42:30.694122+0000","last_active":"2024-03-21T08:42:32.046063+0000","last_peered":"2024-03-21T08:42:32.046063+0000","last_clean":"2024-03-21T08:42:32.046063+0000","last_became_active":"2024-03-21T08:42:30.691718+0000","last_became_peered":"2024-03-21T08:42:30.691718+0000","last_unstale":"2024-03-21T08:42:32.046063+0000","last_undegraded":"2024-03-21T08:42:32.046063+0000","last_fullsized":"2024-03-21T08:42:32.046063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:21:16.354486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688224+0000","last_change":"2024-03-21T08:42:30.692105+0000","last_active":"2024-03-21T08:42:31.688224+0000","last_peered":"2024-03-21T08:42:31.688224+0000","last_clean":"2024-03-21T08:42:31.688224+0000","last_became_active":"2024-03-21T08:42:30.691865+0000","last_became_peered":"2024-03-21T08:42:30.691865+0000","last_unstale":"2024-03-21T08:42:31.688224+0000","last_undegraded":"2024-03-21T08:42:31.688224+0000","last_fullsized":"2024-03-21T08:42:31.688224+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:16:05.453862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046636+0000","last_change":"2024-03-21T08:42:29.683944+0000","last_active":"2024-03-21T08:42:32.046636+0000","last_peered":"2024-03-21T08:42:32.046636+0000","last_clean":"2024-03-21T08:42:32.046636+0000","last_became_active":"2024-03-21T08:42:29.683089+0000","last_became_peered":"2024-03-21T08:42:29.683089+0000","last_unstale":"2024-03-21T08:42:32.046636+0000","last_undegraded":"2024-03-21T08:42:32.046636+0000","last_fullsized":"2024-03-21T08:42:32.046636+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:42:48.211481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047155+0000","last_change":"2024-03-21T08:42:30.700542+0000","last_active":"2024-03-21T08:42:32.047155+0000","last_peered":"2024-03-21T08:42:32.047155+0000","last_clean":"2024-03-21T08:42:32.047155+0000","last_became_active":"2024-03-21T08:42:30.696121+0000","last_became_peered":"2024-03-21T08:42:30.696121+0000","last_unstale":"2024-03-21T08:42:32.047155+0000","last_undegraded":"2024-03-21T08:42:32.047155+0000","last_fullsized":"2024-03-21T08:42:32.047155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:29:30.921384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750995+0000","last_change":"2024-03-21T08:42:29.688290+0000","last_active":"2024-03-21T08:42:31.750995+0000","last_peered":"2024-03-21T08:42:31.750995+0000","last_clean":"2024-03-21T08:42:31.750995+0000","last_became_active":"2024-03-21T08:42:29.687997+0000","last_became_peered":"2024-03-21T08:42:29.687997+0000","last_unstale":"2024-03-21T08:42:31.750995+0000","last_undegraded":"2024-03-21T08:42:31.750995+0000","last_fullsized":"2024-03-21T08:42:31.750995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:11.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046694+0000","last_change":"2024-03-21T08:42:30.692448+0000","last_active":"2024-03-21T08:42:32.046694+0000","last_peered":"2024-03-21T08:42:32.046694+0000","last_clean":"2024-03-21T08:42:32.046694+0000","last_became_active":"2024-03-21T08:42:30.692149+0000","last_became_peered":"2024-03-21T08:42:30.692149+0000","last_unstale":"2024-03-21T08:42:32.046694+0000","last_undegraded":"2024-03-21T08:42:32.046694+0000","last_fullsized":"2024-03-21T08:42:32.046694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:55:15.669183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.047168+0000","last_change":"2024-03-21T08:42:29.683368+0000","last_active":"2024-03-21T08:42:32.047168+0000","last_peered":"2024-03-21T08:42:32.047168+0000","last_clean":"2024-03-21T08:42:32.047168+0000","last_became_active":"2024-03-21T08:42:29.683095+0000","last_became_peered":"2024-03-21T08:42:29.683095+0000","last_unstale":"2024-03-21T08:42:32.047168+0000","last_undegraded":"2024-03-21T08:42:32.047168+0000","last_fullsized":"2024-03-21T08:42:32.047168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:38.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046766+0000","last_change":"2024-03-21T08:42:30.704171+0000","last_active":"2024-03-21T08:42:32.046766+0000","last_peered":"2024-03-21T08:42:32.046766+0000","last_clean":"2024-03-21T08:42:32.046766+0000","last_became_active":"2024-03-21T08:42:30.703815+0000","last_became_peered":"2024-03-21T08:42:30.703815+0000","last_unstale":"2024-03-21T08:42:32.046766+0000","last_undegraded":"2024-03-21T08:42:32.046766+0000","last_fullsized":"2024-03-21T08:42:32.046766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:38:41.370318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046530+0000","last_change":"2024-03-21T08:42:29.687300+0000","last_active":"2024-03-21T08:42:32.046530+0000","last_peered":"2024-03-21T08:42:32.046530+0000","last_clean":"2024-03-21T08:42:32.046530+0000","last_became_active":"2024-03-21T08:42:29.687010+0000","last_became_peered":"2024-03-21T08:42:29.687010+0000","last_unstale":"2024-03-21T08:42:32.046530+0000","last_undegraded":"2024-03-21T08:42:32.046530+0000","last_fullsized":"2024-03-21T08:42:32.046530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:58:11.200847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750892+0000","last_change":"2024-03-21T08:42:30.692975+0000","last_active":"2024-03-21T08:42:31.750892+0000","last_peered":"2024-03-21T08:42:31.750892+0000","last_clean":"2024-03-21T08:42:31.750892+0000","last_became_active":"2024-03-21T08:42:30.692636+0000","last_became_peered":"2024-03-21T08:42:30.692636+0000","last_unstale":"2024-03-21T08:42:31.750892+0000","last_undegraded":"2024-03-21T08:42:31.750892+0000","last_fullsized":"2024-03-21T08:42:31.750892+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:55:15.669183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687279+0000","last_change":"2024-03-21T08:42:29.680829+0000","last_active":"2024-03-21T08:42:31.687279+0000","last_peered":"2024-03-21T08:42:31.687279+0000","last_clean":"2024-03-21T08:42:31.687279+0000","last_became_active":"2024-03-21T08:42:29.680539+0000","last_became_peered":"2024-03-21T08:42:29.680539+0000","last_unstale":"2024-03-21T08:42:31.687279+0000","last_undegraded":"2024-03-21T08:42:31.687279+0000","last_fullsized":"2024-03-21T08:42:31.687279+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:45.341543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687157+0000","last_change":"2024-03-21T08:42:30.693765+0000","last_active":"2024-03-21T08:42:31.687157+0000","last_peered":"2024-03-21T08:42:31.687157+0000","last_clean":"2024-03-21T08:42:31.687157+0000","last_became_active":"2024-03-21T08:42:30.693460+0000","last_became_peered":"2024-03-21T08:42:30.693460+0000","last_unstale":"2024-03-21T08:42:31.687157+0000","last_undegraded":"2024-03-21T08:42:31.687157+0000","last_fullsized":"2024-03-21T08:42:31.687157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:46:40.800907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687619+0000","last_change":"2024-03-21T08:42:29.689467+0000","last_active":"2024-03-21T08:42:31.687619+0000","last_peered":"2024-03-21T08:42:31.687619+0000","last_clean":"2024-03-21T08:42:31.687619+0000","last_became_active":"2024-03-21T08:42:29.688797+0000","last_became_peered":"2024-03-21T08:42:29.688797+0000","last_unstale":"2024-03-21T08:42:31.687619+0000","last_undegraded":"2024-03-21T08:42:31.687619+0000","last_fullsized":"2024-03-21T08:42:31.687619+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:38:24.494075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046343+0000","last_change":"2024-03-21T08:42:30.680880+0000","last_active":"2024-03-21T08:42:32.046343+0000","last_peered":"2024-03-21T08:42:32.046343+0000","last_clean":"2024-03-21T08:42:32.046343+0000","last_became_active":"2024-03-21T08:42:30.680614+0000","last_became_peered":"2024-03-21T08:42:30.680614+0000","last_unstale":"2024-03-21T08:42:32.046343+0000","last_undegraded":"2024-03-21T08:42:32.046343+0000","last_fullsized":"2024-03-21T08:42:32.046343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:33:36.238601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046141+0000","last_change":"2024-03-21T08:42:29.688043+0000","last_active":"2024-03-21T08:42:32.046141+0000","last_peered":"2024-03-21T08:42:32.046141+0000","last_clean":"2024-03-21T08:42:32.046141+0000","last_became_active":"2024-03-21T08:42:29.686991+0000","last_became_peered":"2024-03-21T08:42:29.686991+0000","last_unstale":"2024-03-21T08:42:32.046141+0000","last_undegraded":"2024-03-21T08:42:32.046141+0000","last_fullsized":"2024-03-21T08:42:32.046141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:03.893918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046640+0000","last_change":"2024-03-21T08:42:30.702252+0000","last_active":"2024-03-21T08:42:32.046640+0000","last_peered":"2024-03-21T08:42:32.046640+0000","last_clean":"2024-03-21T08:42:32.046640+0000","last_became_active":"2024-03-21T08:42:30.701982+0000","last_became_peered":"2024-03-21T08:42:30.701982+0000","last_unstale":"2024-03-21T08:42:32.046640+0000","last_undegraded":"2024-03-21T08:42:32.046640+0000","last_fullsized":"2024-03-21T08:42:32.046640+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:28:41.046722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045717+0000","last_change":"2024-03-21T08:42:29.688835+0000","last_active":"2024-03-21T08:42:32.045717+0000","last_peered":"2024-03-21T08:42:32.045717+0000","last_clean":"2024-03-21T08:42:32.045717+0000","last_became_active":"2024-03-21T08:42:29.688547+0000","last_became_peered":"2024-03-21T08:42:29.688547+0000","last_unstale":"2024-03-21T08:42:32.045717+0000","last_undegraded":"2024-03-21T08:42:32.045717+0000","last_fullsized":"2024-03-21T08:42:32.045717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:00:31.687764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687081+0000","last_change":"2024-03-21T08:42:30.678062+0000","last_active":"2024-03-21T08:42:31.687081+0000","last_peered":"2024-03-21T08:42:31.687081+0000","last_clean":"2024-03-21T08:42:31.687081+0000","last_became_active":"2024-03-21T08:42:30.676930+0000","last_became_peered":"2024-03-21T08:42:30.676930+0000","last_unstale":"2024-03-21T08:42:31.687081+0000","last_undegraded":"2024-03-21T08:42:31.687081+0000","last_fullsized":"2024-03-21T08:42:31.687081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:39.027444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046818+0000","last_change":"2024-03-21T08:42:29.689147+0000","last_active":"2024-03-21T08:42:32.046818+0000","last_peered":"2024-03-21T08:42:32.046818+0000","last_clean":"2024-03-21T08:42:32.046818+0000","last_became_active":"2024-03-21T08:42:29.688858+0000","last_became_peered":"2024-03-21T08:42:29.688858+0000","last_unstale":"2024-03-21T08:42:32.046818+0000","last_undegraded":"2024-03-21T08:42:32.046818+0000","last_fullsized":"2024-03-21T08:42:32.046818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:04:54.658152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688049+0000","last_change":"2024-03-21T08:42:30.698953+0000","last_active":"2024-03-21T08:42:31.688049+0000","last_peered":"2024-03-21T08:42:31.688049+0000","last_clean":"2024-03-21T08:42:31.688049+0000","last_became_active":"2024-03-21T08:42:30.698585+0000","last_became_peered":"2024-03-21T08:42:30.698585+0000","last_unstale":"2024-03-21T08:42:31.688049+0000","last_undegraded":"2024-03-21T08:42:31.688049+0000","last_fullsized":"2024-03-21T08:42:31.688049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:28:30.269037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687406+0000","last_change":"2024-03-21T08:42:29.690665+0000","last_active":"2024-03-21T08:42:31.687406+0000","last_peered":"2024-03-21T08:42:31.687406+0000","last_clean":"2024-03-21T08:42:31.687406+0000","last_became_active":"2024-03-21T08:42:29.690396+0000","last_became_peered":"2024-03-21T08:42:29.690396+0000","last_unstale":"2024-03-21T08:42:31.687406+0000","last_undegraded":"2024-03-21T08:42:31.687406+0000","last_fullsized":"2024-03-21T08:42:31.687406+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:26:15.673658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751255+0000","last_change":"2024-03-21T08:42:30.697681+0000","last_active":"2024-03-21T08:42:31.751255+0000","last_peered":"2024-03-21T08:42:31.751255+0000","last_clean":"2024-03-21T08:42:31.751255+0000","last_became_active":"2024-03-21T08:42:30.697400+0000","last_became_peered":"2024-03-21T08:42:30.697400+0000","last_unstale":"2024-03-21T08:42:31.751255+0000","last_undegraded":"2024-03-21T08:42:31.751255+0000","last_fullsized":"2024-03-21T08:42:31.751255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:32:42.990151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687899+0000","last_change":"2024-03-21T08:42:29.681244+0000","last_active":"2024-03-21T08:42:31.687899+0000","last_peered":"2024-03-21T08:42:31.687899+0000","last_clean":"2024-03-21T08:42:31.687899+0000","last_became_active":"2024-03-21T08:42:29.680101+0000","last_became_peered":"2024-03-21T08:42:29.680101+0000","last_unstale":"2024-03-21T08:42:31.687899+0000","last_undegraded":"2024-03-21T08:42:31.687899+0000","last_fullsized":"2024-03-21T08:42:31.687899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:04:54.658152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046863+0000","last_change":"2024-03-21T08:42:30.684751+0000","last_active":"2024-03-21T08:42:32.046863+0000","last_peered":"2024-03-21T08:42:32.046863+0000","last_clean":"2024-03-21T08:42:32.046863+0000","last_became_active":"2024-03-21T08:42:30.683841+0000","last_became_peered":"2024-03-21T08:42:30.683841+0000","last_unstale":"2024-03-21T08:42:32.046863+0000","last_undegraded":"2024-03-21T08:42:32.046863+0000","last_fullsized":"2024-03-21T08:42:32.046863+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:21:12.042436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046596+0000","last_change":"2024-03-21T08:42:29.689562+0000","last_active":"2024-03-21T08:42:32.046596+0000","last_peered":"2024-03-21T08:42:32.046596+0000","last_clean":"2024-03-21T08:42:32.046596+0000","last_became_active":"2024-03-21T08:42:29.687676+0000","last_became_peered":"2024-03-21T08:42:29.687676+0000","last_unstale":"2024-03-21T08:42:32.046596+0000","last_undegraded":"2024-03-21T08:42:32.046596+0000","last_fullsized":"2024-03-21T08:42:32.046596+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:03:39.982655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750777+0000","last_change":"2024-03-21T08:42:30.682253+0000","last_active":"2024-03-21T08:42:31.750777+0000","last_peered":"2024-03-21T08:42:31.750777+0000","last_clean":"2024-03-21T08:42:31.750777+0000","last_became_active":"2024-03-21T08:42:30.681976+0000","last_became_peered":"2024-03-21T08:42:30.681976+0000","last_unstale":"2024-03-21T08:42:31.750777+0000","last_undegraded":"2024-03-21T08:42:31.750777+0000","last_fullsized":"2024-03-21T08:42:31.750777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:33:36.238601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046694+0000","last_change":"2024-03-21T08:42:29.690366+0000","last_active":"2024-03-21T08:42:32.046694+0000","last_peered":"2024-03-21T08:42:32.046694+0000","last_clean":"2024-03-21T08:42:32.046694+0000","last_became_active":"2024-03-21T08:42:29.690030+0000","last_became_peered":"2024-03-21T08:42:29.690030+0000","last_unstale":"2024-03-21T08:42:32.046694+0000","last_undegraded":"2024-03-21T08:42:32.046694+0000","last_fullsized":"2024-03-21T08:42:32.046694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:57:52.751505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046344+0000","last_change":"2024-03-21T08:42:30.686858+0000","last_active":"2024-03-21T08:42:32.046344+0000","last_peered":"2024-03-21T08:42:32.046344+0000","last_clean":"2024-03-21T08:42:32.046344+0000","last_became_active":"2024-03-21T08:42:30.686587+0000","last_became_peered":"2024-03-21T08:42:30.686587+0000","last_unstale":"2024-03-21T08:42:32.046344+0000","last_undegraded":"2024-03-21T08:42:32.046344+0000","last_fullsized":"2024-03-21T08:42:32.046344+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:11:21.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046657+0000","last_change":"2024-03-21T08:42:29.687057+0000","last_active":"2024-03-21T08:42:32.046657+0000","last_peered":"2024-03-21T08:42:32.046657+0000","last_clean":"2024-03-21T08:42:32.046657+0000","last_became_active":"2024-03-21T08:42:29.686782+0000","last_became_peered":"2024-03-21T08:42:29.686782+0000","last_unstale":"2024-03-21T08:42:32.046657+0000","last_undegraded":"2024-03-21T08:42:32.046657+0000","last_fullsized":"2024-03-21T08:42:32.046657+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:10:13.487644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687512+0000","last_change":"2024-03-21T08:42:30.695584+0000","last_active":"2024-03-21T08:42:31.687512+0000","last_peered":"2024-03-21T08:42:31.687512+0000","last_clean":"2024-03-21T08:42:31.687512+0000","last_became_active":"2024-03-21T08:42:30.695232+0000","last_became_peered":"2024-03-21T08:42:30.695232+0000","last_unstale":"2024-03-21T08:42:31.687512+0000","last_undegraded":"2024-03-21T08:42:31.687512+0000","last_fullsized":"2024-03-21T08:42:31.687512+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:16:30.082762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045569+0000","last_change":"2024-03-21T08:42:29.685401+0000","last_active":"2024-03-21T08:42:32.045569+0000","last_peered":"2024-03-21T08:42:32.045569+0000","last_clean":"2024-03-21T08:42:32.045569+0000","last_became_active":"2024-03-21T08:42:29.683391+0000","last_became_peered":"2024-03-21T08:42:29.683391+0000","last_unstale":"2024-03-21T08:42:32.045569+0000","last_undegraded":"2024-03-21T08:42:32.045569+0000","last_fullsized":"2024-03-21T08:42:32.045569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:54:42.529477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046572+0000","last_change":"2024-03-21T08:42:30.696914+0000","last_active":"2024-03-21T08:42:32.046572+0000","last_peered":"2024-03-21T08:42:32.046572+0000","last_clean":"2024-03-21T08:42:32.046572+0000","last_became_active":"2024-03-21T08:42:30.696632+0000","last_became_peered":"2024-03-21T08:42:30.696632+0000","last_unstale":"2024-03-21T08:42:32.046572+0000","last_undegraded":"2024-03-21T08:42:32.046572+0000","last_fullsized":"2024-03-21T08:42:32.046572+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:58:57.974782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686893+0000","last_change":"2024-03-21T08:42:29.689164+0000","last_active":"2024-03-21T08:42:31.686893+0000","last_peered":"2024-03-21T08:42:31.686893+0000","last_clean":"2024-03-21T08:42:31.686893+0000","last_became_active":"2024-03-21T08:42:29.688812+0000","last_became_peered":"2024-03-21T08:42:29.688812+0000","last_unstale":"2024-03-21T08:42:31.686893+0000","last_undegraded":"2024-03-21T08:42:31.686893+0000","last_fullsized":"2024-03-21T08:42:31.686893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:03:38.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046583+0000","last_change":"2024-03-21T08:42:30.701754+0000","last_active":"2024-03-21T08:42:32.046583+0000","last_peered":"2024-03-21T08:42:32.046583+0000","last_clean":"2024-03-21T08:42:32.046583+0000","last_became_active":"2024-03-21T08:42:30.701476+0000","last_became_peered":"2024-03-21T08:42:30.701476+0000","last_unstale":"2024-03-21T08:42:32.046583+0000","last_undegraded":"2024-03-21T08:42:32.046583+0000","last_fullsized":"2024-03-21T08:42:32.046583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:32.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687642+0000","last_change":"2024-03-21T08:42:29.683970+0000","last_active":"2024-03-21T08:42:31.687642+0000","last_peered":"2024-03-21T08:42:31.687642+0000","last_clean":"2024-03-21T08:42:31.687642+0000","last_became_active":"2024-03-21T08:42:29.683655+0000","last_became_peered":"2024-03-21T08:42:29.683655+0000","last_unstale":"2024-03-21T08:42:31.687642+0000","last_undegraded":"2024-03-21T08:42:31.687642+0000","last_fullsized":"2024-03-21T08:42:31.687642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:46:36.488242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687897+0000","last_change":"2024-03-21T08:42:30.693462+0000","last_active":"2024-03-21T08:42:31.687897+0000","last_peered":"2024-03-21T08:42:31.687897+0000","last_clean":"2024-03-21T08:42:31.687897+0000","last_became_active":"2024-03-21T08:42:30.693030+0000","last_became_peered":"2024-03-21T08:42:30.693030+0000","last_unstale":"2024-03-21T08:42:31.687897+0000","last_undegraded":"2024-03-21T08:42:31.687897+0000","last_fullsized":"2024-03-21T08:42:31.687897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:18.960768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046668+0000","last_change":"2024-03-21T08:42:29.683875+0000","last_active":"2024-03-21T08:42:32.046668+0000","last_peered":"2024-03-21T08:42:32.046668+0000","last_clean":"2024-03-21T08:42:32.046668+0000","last_became_active":"2024-03-21T08:42:29.683393+0000","last_became_peered":"2024-03-21T08:42:29.683393+0000","last_unstale":"2024-03-21T08:42:32.046668+0000","last_undegraded":"2024-03-21T08:42:32.046668+0000","last_fullsized":"2024-03-21T08:42:32.046668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:54:19.261503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046630+0000","last_change":"2024-03-21T08:42:30.689908+0000","last_active":"2024-03-21T08:42:32.046630+0000","last_peered":"2024-03-21T08:42:32.046630+0000","last_clean":"2024-03-21T08:42:32.046630+0000","last_became_active":"2024-03-21T08:42:30.689649+0000","last_became_peered":"2024-03-21T08:42:30.689649+0000","last_unstale":"2024-03-21T08:42:32.046630+0000","last_undegraded":"2024-03-21T08:42:32.046630+0000","last_fullsized":"2024-03-21T08:42:32.046630+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:33:21.214239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686906+0000","last_change":"2024-03-21T08:42:29.685315+0000","last_active":"2024-03-21T08:42:31.686906+0000","last_peered":"2024-03-21T08:42:31.686906+0000","last_clean":"2024-03-21T08:42:31.686906+0000","last_became_active":"2024-03-21T08:42:29.684654+0000","last_became_peered":"2024-03-21T08:42:29.684654+0000","last_unstale":"2024-03-21T08:42:31.686906+0000","last_undegraded":"2024-03-21T08:42:31.686906+0000","last_fullsized":"2024-03-21T08:42:31.686906+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:28:45.351752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687118+0000","last_change":"2024-03-21T08:42:30.680532+0000","last_active":"2024-03-21T08:42:31.687118+0000","last_peered":"2024-03-21T08:42:31.687118+0000","last_clean":"2024-03-21T08:42:31.687118+0000","last_became_active":"2024-03-21T08:42:30.680074+0000","last_became_peered":"2024-03-21T08:42:30.680074+0000","last_unstale":"2024-03-21T08:42:31.687118+0000","last_undegraded":"2024-03-21T08:42:31.687118+0000","last_fullsized":"2024-03-21T08:42:31.687118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:38:25.501198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046948+0000","last_change":"2024-03-21T08:42:29.682271+0000","last_active":"2024-03-21T08:42:32.046948+0000","last_peered":"2024-03-21T08:42:32.046948+0000","last_clean":"2024-03-21T08:42:32.046948+0000","last_became_active":"2024-03-21T08:42:29.681820+0000","last_became_peered":"2024-03-21T08:42:29.681820+0000","last_unstale":"2024-03-21T08:42:32.046948+0000","last_undegraded":"2024-03-21T08:42:32.046948+0000","last_fullsized":"2024-03-21T08:42:32.046948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:39:54.299526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687052+0000","last_change":"2024-03-21T08:42:30.698905+0000","last_active":"2024-03-21T08:42:31.687052+0000","last_peered":"2024-03-21T08:42:31.687052+0000","last_clean":"2024-03-21T08:42:31.687052+0000","last_became_active":"2024-03-21T08:42:30.698438+0000","last_became_peered":"2024-03-21T08:42:30.698438+0000","last_unstale":"2024-03-21T08:42:31.687052+0000","last_undegraded":"2024-03-21T08:42:31.687052+0000","last_fullsized":"2024-03-21T08:42:31.687052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:04:31.289373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686603+0000","last_change":"2024-03-21T08:42:29.681387+0000","last_active":"2024-03-21T08:42:31.686603+0000","last_peered":"2024-03-21T08:42:31.686603+0000","last_clean":"2024-03-21T08:42:31.686603+0000","last_became_active":"2024-03-21T08:42:29.681045+0000","last_became_peered":"2024-03-21T08:42:29.681045+0000","last_unstale":"2024-03-21T08:42:31.686603+0000","last_undegraded":"2024-03-21T08:42:31.686603+0000","last_fullsized":"2024-03-21T08:42:31.686603+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:54.348736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750787+0000","last_change":"2024-03-21T08:42:30.696906+0000","last_active":"2024-03-21T08:42:31.750787+0000","last_peered":"2024-03-21T08:42:31.750787+0000","last_clean":"2024-03-21T08:42:31.750787+0000","last_became_active":"2024-03-21T08:42:30.696649+0000","last_became_peered":"2024-03-21T08:42:30.696649+0000","last_unstale":"2024-03-21T08:42:31.750787+0000","last_undegraded":"2024-03-21T08:42:31.750787+0000","last_fullsized":"2024-03-21T08:42:31.750787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:05.195688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686426+0000","last_change":"2024-03-21T08:42:29.686227+0000","last_active":"2024-03-21T08:42:31.686426+0000","last_peered":"2024-03-21T08:42:31.686426+0000","last_clean":"2024-03-21T08:42:31.686426+0000","last_became_active":"2024-03-21T08:42:29.685952+0000","last_became_peered":"2024-03-21T08:42:29.685952+0000","last_unstale":"2024-03-21T08:42:31.686426+0000","last_undegraded":"2024-03-21T08:42:31.686426+0000","last_fullsized":"2024-03-21T08:42:31.686426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:06:27.633396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688555+0000","last_change":"2024-03-21T08:42:30.687101+0000","last_active":"2024-03-21T08:42:31.688555+0000","last_peered":"2024-03-21T08:42:31.688555+0000","last_clean":"2024-03-21T08:42:31.688555+0000","last_became_active":"2024-03-21T08:42:30.686821+0000","last_became_peered":"2024-03-21T08:42:30.686821+0000","last_unstale":"2024-03-21T08:42:31.688555+0000","last_undegraded":"2024-03-21T08:42:31.688555+0000","last_fullsized":"2024-03-21T08:42:31.688555+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:03:38.474542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688727+0000","last_change":"2024-03-21T08:42:29.689729+0000","last_active":"2024-03-21T08:42:31.688727+0000","last_peered":"2024-03-21T08:42:31.688727+0000","last_clean":"2024-03-21T08:42:31.688727+0000","last_became_active":"2024-03-21T08:42:29.689417+0000","last_became_peered":"2024-03-21T08:42:29.689417+0000","last_unstale":"2024-03-21T08:42:31.688727+0000","last_undegraded":"2024-03-21T08:42:31.688727+0000","last_fullsized":"2024-03-21T08:42:31.688727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:16:03.369097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046786+0000","last_change":"2024-03-21T08:42:30.696977+0000","last_active":"2024-03-21T08:42:32.046786+0000","last_peered":"2024-03-21T08:42:32.046786+0000","last_clean":"2024-03-21T08:42:32.046786+0000","last_became_active":"2024-03-21T08:42:30.696683+0000","last_became_peered":"2024-03-21T08:42:30.696683+0000","last_unstale":"2024-03-21T08:42:32.046786+0000","last_undegraded":"2024-03-21T08:42:32.046786+0000","last_fullsized":"2024-03-21T08:42:32.046786+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:35.440529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.686313+0000","last_change":"2024-03-21T08:42:29.681441+0000","last_active":"2024-03-21T08:42:31.686313+0000","last_peered":"2024-03-21T08:42:31.686313+0000","last_clean":"2024-03-21T08:42:31.686313+0000","last_became_active":"2024-03-21T08:42:29.680958+0000","last_became_peered":"2024-03-21T08:42:29.680958+0000","last_unstale":"2024-03-21T08:42:31.686313+0000","last_undegraded":"2024-03-21T08:42:31.686313+0000","last_fullsized":"2024-03-21T08:42:31.686313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:29:17.224372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046923+0000","last_change":"2024-03-21T08:42:30.690979+0000","last_active":"2024-03-21T08:42:32.046923+0000","last_peered":"2024-03-21T08:42:32.046923+0000","last_clean":"2024-03-21T08:42:32.046923+0000","last_became_active":"2024-03-21T08:42:30.690659+0000","last_became_peered":"2024-03-21T08:42:30.690659+0000","last_unstale":"2024-03-21T08:42:32.046923+0000","last_undegraded":"2024-03-21T08:42:32.046923+0000","last_fullsized":"2024-03-21T08:42:32.046923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:24:58.246991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687767+0000","last_change":"2024-03-21T08:42:29.686849+0000","last_active":"2024-03-21T08:42:31.687767+0000","last_peered":"2024-03-21T08:42:31.687767+0000","last_clean":"2024-03-21T08:42:31.687767+0000","last_became_active":"2024-03-21T08:42:29.686581+0000","last_became_peered":"2024-03-21T08:42:29.686581+0000","last_unstale":"2024-03-21T08:42:31.687767+0000","last_undegraded":"2024-03-21T08:42:31.687767+0000","last_fullsized":"2024-03-21T08:42:31.687767+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:01.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688218+0000","last_change":"2024-03-21T08:42:30.699470+0000","last_active":"2024-03-21T08:42:31.688218+0000","last_peered":"2024-03-21T08:42:31.688218+0000","last_clean":"2024-03-21T08:42:31.688218+0000","last_became_active":"2024-03-21T08:42:30.699169+0000","last_became_peered":"2024-03-21T08:42:30.699169+0000","last_unstale":"2024-03-21T08:42:31.688218+0000","last_undegraded":"2024-03-21T08:42:31.688218+0000","last_fullsized":"2024-03-21T08:42:31.688218+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:02:36.190026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687019+0000","last_change":"2024-03-21T08:42:29.680363+0000","last_active":"2024-03-21T08:42:31.687019+0000","last_peered":"2024-03-21T08:42:31.687019+0000","last_clean":"2024-03-21T08:42:31.687019+0000","last_became_active":"2024-03-21T08:42:29.680096+0000","last_became_peered":"2024-03-21T08:42:29.680096+0000","last_unstale":"2024-03-21T08:42:31.687019+0000","last_undegraded":"2024-03-21T08:42:31.687019+0000","last_fullsized":"2024-03-21T08:42:31.687019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:30:03.838556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046835+0000","last_change":"2024-03-21T08:42:30.685856+0000","last_active":"2024-03-21T08:42:32.046835+0000","last_peered":"2024-03-21T08:42:32.046835+0000","last_clean":"2024-03-21T08:42:32.046835+0000","last_became_active":"2024-03-21T08:42:30.685578+0000","last_became_peered":"2024-03-21T08:42:30.685578+0000","last_unstale":"2024-03-21T08:42:32.046835+0000","last_undegraded":"2024-03-21T08:42:32.046835+0000","last_fullsized":"2024-03-21T08:42:32.046835+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:40:18.960768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750952+0000","last_change":"2024-03-21T08:42:29.689582+0000","last_active":"2024-03-21T08:42:31.750952+0000","last_peered":"2024-03-21T08:42:31.750952+0000","last_clean":"2024-03-21T08:42:31.750952+0000","last_became_active":"2024-03-21T08:42:29.689346+0000","last_became_peered":"2024-03-21T08:42:29.689346+0000","last_unstale":"2024-03-21T08:42:31.750952+0000","last_undegraded":"2024-03-21T08:42:31.750952+0000","last_fullsized":"2024-03-21T08:42:31.750952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:00:31.687764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.751089+0000","last_change":"2024-03-21T08:42:30.691466+0000","last_active":"2024-03-21T08:42:31.751089+0000","last_peered":"2024-03-21T08:42:31.751089+0000","last_clean":"2024-03-21T08:42:31.751089+0000","last_became_active":"2024-03-21T08:42:30.688723+0000","last_became_peered":"2024-03-21T08:42:30.688723+0000","last_unstale":"2024-03-21T08:42:31.751089+0000","last_undegraded":"2024-03-21T08:42:31.751089+0000","last_fullsized":"2024-03-21T08:42:31.751089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:51:06.301613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.750901+0000","last_change":"2024-03-21T08:42:29.685783+0000","last_active":"2024-03-21T08:42:31.750901+0000","last_peered":"2024-03-21T08:42:31.750901+0000","last_clean":"2024-03-21T08:42:31.750901+0000","last_became_active":"2024-03-21T08:42:29.685512+0000","last_became_peered":"2024-03-21T08:42:29.685512+0000","last_unstale":"2024-03-21T08:42:31.750901+0000","last_undegraded":"2024-03-21T08:42:31.750901+0000","last_fullsized":"2024-03-21T08:42:31.750901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:40.758395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046075+0000","last_change":"2024-03-21T08:42:30.697765+0000","last_active":"2024-03-21T08:42:32.046075+0000","last_peered":"2024-03-21T08:42:32.046075+0000","last_clean":"2024-03-21T08:42:32.046075+0000","last_became_active":"2024-03-21T08:42:30.695427+0000","last_became_peered":"2024-03-21T08:42:30.695427+0000","last_unstale":"2024-03-21T08:42:32.046075+0000","last_undegraded":"2024-03-21T08:42:32.046075+0000","last_fullsized":"2024-03-21T08:42:32.046075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:16:08.486846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045885+0000","last_change":"2024-03-21T08:42:29.681258+0000","last_active":"2024-03-21T08:42:32.045885+0000","last_peered":"2024-03-21T08:42:32.045885+0000","last_clean":"2024-03-21T08:42:32.045885+0000","last_became_active":"2024-03-21T08:42:29.680977+0000","last_became_peered":"2024-03-21T08:42:29.680977+0000","last_unstale":"2024-03-21T08:42:32.045885+0000","last_undegraded":"2024-03-21T08:42:32.045885+0000","last_fullsized":"2024-03-21T08:42:32.045885+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:56:19.956820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046550+0000","last_change":"2024-03-21T08:42:30.679950+0000","last_active":"2024-03-21T08:42:32.046550+0000","last_peered":"2024-03-21T08:42:32.046550+0000","last_clean":"2024-03-21T08:42:32.046550+0000","last_became_active":"2024-03-21T08:42:30.679672+0000","last_became_peered":"2024-03-21T08:42:30.679672+0000","last_unstale":"2024-03-21T08:42:32.046550+0000","last_undegraded":"2024-03-21T08:42:32.046550+0000","last_fullsized":"2024-03-21T08:42:32.046550+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:26:05.195688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.045674+0000","last_change":"2024-03-21T08:42:29.687355+0000","last_active":"2024-03-21T08:42:32.045674+0000","last_peered":"2024-03-21T08:42:32.045674+0000","last_clean":"2024-03-21T08:42:32.045674+0000","last_became_active":"2024-03-21T08:42:29.687072+0000","last_became_peered":"2024-03-21T08:42:29.687072+0000","last_unstale":"2024-03-21T08:42:32.045674+0000","last_undegraded":"2024-03-21T08:42:32.045674+0000","last_fullsized":"2024-03-21T08:42:32.045674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:39:40.758395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688114+0000","last_change":"2024-03-21T08:42:30.699318+0000","last_active":"2024-03-21T08:42:31.688114+0000","last_peered":"2024-03-21T08:42:31.688114+0000","last_clean":"2024-03-21T08:42:31.688114+0000","last_became_active":"2024-03-21T08:42:30.699015+0000","last_became_peered":"2024-03-21T08:42:30.699015+0000","last_unstale":"2024-03-21T08:42:31.688114+0000","last_undegraded":"2024-03-21T08:42:31.688114+0000","last_fullsized":"2024-03-21T08:42:31.688114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:57:55.777767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:32.046238+0000","last_change":"2024-03-21T08:42:29.685473+0000","last_active":"2024-03-21T08:42:32.046238+0000","last_peered":"2024-03-21T08:42:32.046238+0000","last_clean":"2024-03-21T08:42:32.046238+0000","last_became_active":"2024-03-21T08:42:29.685208+0000","last_became_peered":"2024-03-21T08:42:29.685208+0000","last_unstale":"2024-03-21T08:42:32.046238+0000","last_undegraded":"2024-03-21T08:42:32.046238+0000","last_fullsized":"2024-03-21T08:42:32.046238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:39:45.341543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687622+0000","last_change":"2024-03-21T08:42:30.681206+0000","last_active":"2024-03-21T08:42:31.687622+0000","last_peered":"2024-03-21T08:42:31.687622+0000","last_clean":"2024-03-21T08:42:31.687622+0000","last_became_active":"2024-03-21T08:42:30.680936+0000","last_became_peered":"2024-03-21T08:42:30.680936+0000","last_unstale":"2024-03-21T08:42:31.687622+0000","last_undegraded":"2024-03-21T08:42:31.687622+0000","last_fullsized":"2024-03-21T08:42:31.687622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:29.660854+0000","last_clean_scrub_stamp":"2024-03-21T08:42:29.660854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:28:41.046722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.688329+0000","last_change":"2024-03-21T08:42:29.692572+0000","last_active":"2024-03-21T08:42:31.688329+0000","last_peered":"2024-03-21T08:42:31.688329+0000","last_clean":"2024-03-21T08:42:31.688329+0000","last_became_active":"2024-03-21T08:42:29.692230+0000","last_became_peered":"2024-03-21T08:42:29.692230+0000","last_unstale":"2024-03-21T08:42:31.688329+0000","last_undegraded":"2024-03-21T08:42:31.688329+0000","last_fullsized":"2024-03-21T08:42:31.688329+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:42:28.653731+0000","last_clean_scrub_stamp":"2024-03-21T08:42:28.653731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:03.893918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-21T08:42:31.687310+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:42:31.687310+0000","last_peered":"2024-03-21T08:42:31.687310+0000","last_clean":"2024-03-21T08:42:31.687310+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:42:31.687310+0000","last_undegraded":"2024-03-21T08:42:31.687310+0000","last_fullsized":"2024-03-21T08:42:31.687310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1052672,"data_stored":784071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":192603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509463,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":192091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":446464,"data_stored":192111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542168,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542168,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":193667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1040384,"data_stored":782423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:42:39.791 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-21T08:42:39.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-21T08:42:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.971+0000 7f5deb800640 1 Processor -- start 2024-03-21T08:42:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.971+0000 7f5deb800640 1 -- start start 2024-03-21T08:42:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5deb800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5de415bd20 0x7f5de40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5deb800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de40b0e80 0x7f5de40ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5deb800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de415fba0 con 0x7f5de40cea00 2024-03-21T08:42:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5deb800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de408bfd0 con 0x7f5de415bd20 2024-03-21T08:42:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5deb800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de40cee50 con 0x7f5de40b0e80 2024-03-21T08:42:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5de415bd20 0x7f5de40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5de40cea00 0x7f5de40b19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54524/0 (socket says 172.21.15.28:54524) 2024-03-21T08:42:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de9800640 1 -- 172.21.15.28:0/3278404620 learned_addr learned my addr 172.21.15.28:0/3278404620 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5ddbe00640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de40b0e80 0x7f5de40ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de8e00640 1 -- 172.21.15.28:0/3278404620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de40b0e80 msgr2=0x7f5de40ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de3e00640 1 -- 172.21.15.28:0/3278404620 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589593425 0 0) 0x7f5de415fba0 con 0x7f5de40cea00 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de8e00640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de40b0e80 0x7f5de40ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de8e00640 1 -- 172.21.15.28:0/3278404620 >> v1:172.21.15.28:6789/0 conn(0x7f5de40cea00 legacy=0x7f5de40b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.975+0000 7f5de8e00640 1 -- 172.21.15.28:0/3278404620 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de415fe80 con 0x7f5de415bd20 2024-03-21T08:42:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5de8e00640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de415bd20 0x7f5de40b2c40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40097c0 tx=0x7f5dd401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d20e8eca2bcddc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5de3e00640 1 -- 172.21.15.28:0/3278404620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd40b0070 con 0x7f5de415bd20 2024-03-21T08:42:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5de3e00640 1 -- 172.21.15.28:0/3278404620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd40ab600 con 0x7f5de415bd20 2024-03-21T08:42:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5de3e00640 1 -- 172.21.15.28:0/3278404620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd40ab9d0 con 0x7f5de415bd20 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- 172.21.15.28:0/3278404620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de415bd20 msgr2=0x7f5de40b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de415bd20 0x7f5de40b2c40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40097c0 tx=0x7f5dd401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- 172.21.15.28:0/3278404620 shutdown_connections 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de40b0e80 0x7f5de40ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 --2- 172.21.15.28:0/3278404620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de415bd20 0x7f5de40b2c40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- 172.21.15.28:0/3278404620 >> 172.21.15.28:0/3278404620 conn(0x7f5de40d1ab0 msgr2=0x7f5de40d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- 172.21.15.28:0/3278404620 shutdown_connections 2024-03-21T08:42:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- 172.21.15.28:0/3278404620 wait complete. 2024-03-21T08:42:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 Processor -- start 2024-03-21T08:42:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.979+0000 7f5deb800640 1 -- start start 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 0x7f5de406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de415bd20 0x7f5de41578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de40cc6a0 con 0x7f5de40b0e80 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de40cc520 con 0x7f5de40cea00 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de415c100 con 0x7f5de415bd20 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43254/0 (socket says 172.21.15.28:43254) 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 -- 172.21.15.28:0/2439418802 learned_addr learned my addr 172.21.15.28:0/2439418802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5ddbe00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 0x7f5de406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de9800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de415bd20 0x7f5de41578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 -- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de415bd20 msgr2=0x7f5de41578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de415bd20 0x7f5de41578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 -- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 msgr2=0x7f5de406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 0x7f5de406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 -- 172.21.15.28:0/2439418802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de406cbd0 con 0x7f5de40b0e80 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5ddbe00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 0x7f5de406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de8e00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40ae040 tx=0x7f5dd401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd40b0050 con 0x7f5de40b0e80 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd40bc070 con 0x7f5de40b0e80 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de406ceb0 con 0x7f5de40b0e80 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd40b5850 con 0x7f5de40b0e80 2024-03-21T08:42:39.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de41570a0 con 0x7f5de40b0e80 2024-03-21T08:42:39.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de404f4f0 con 0x7f5de40b0e80 2024-03-21T08:42:39.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.983+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5dd40b59f0 con 0x7f5de40b0e80 2024-03-21T08:42:39.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.987+0000 7f5de1600640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 0x7f5dc0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.987+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5dd4121260 con 0x7f5de40b0e80 2024-03-21T08:42:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.987+0000 7f5ddbe00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 0x7f5dc0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.987+0000 7f5ddbe00640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 0x7f5dc0069fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5de4159210 tx=0x7f5dcc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:39.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:39.987+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5dd4016090 con 0x7f5de40b0e80 2024-03-21T08:42:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.207+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5de40d2830 con 0x7f5de40b0e80 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5de1600640 1 -- 172.21.15.28:0/2439418802 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5dd40f1ac0 con 0x7f5de40b0e80 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:42:40.213 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-21T08:42:40.214 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:42:40.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 msgr2=0x7f5dc0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 0x7f5dc0069fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5de4159210 tx=0x7f5dcc008040 comp rx=0 tx=0).stop 2024-03-21T08:42:40.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 msgr2=0x7f5de4151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40ae040 tx=0x7f5dd401ace0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 shutdown_connections 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5dc0067b20 0x7f5dc0069fe0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5de415bd20 0x7f5de41578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5de40cea00 0x7f5de406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 --2- 172.21.15.28:0/2439418802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de40b0e80 0x7f5de4151af0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 >> 172.21.15.28:0/2439418802 conn(0x7f5de40d1ab0 msgr2=0x7f5de408afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 shutdown_connections 2024-03-21T08:42:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.211+0000 7f5deb800640 1 -- 172.21.15.28:0/2439418802 wait complete. 2024-03-21T08:42:40.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 Processor -- start 2024-03-21T08:42:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 -- start start 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c815f260 0x7f82c80cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f82c80cd5d0 0x7f82c8099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c809a0c0 0x7f82c80cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c80c5530 con 0x7f82c815f260 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c80cdb50 con 0x7f82c80cd5d0 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82cdc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80cde30 con 0x7f82c809a0c0 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c815f260 0x7f82c80cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c815f260 0x7f82c80cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38158/0 (socket says 172.21.15.28:38158) 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 -- 172.21.15.28:0/2740972637 learned_addr learned my addr 172.21.15.28:0/2740972637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c7400640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c809a0c0 0x7f82c80cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6000640 1 --2- 172.21.15.28:0/2740972637 >> v2:172.21.15.151:3300/0 conn(0x7f82c80cd5d0 0x7f82c8099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 -- 172.21.15.28:0/2740972637 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c809a0c0 msgr2=0x7f82c80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c809a0c0 0x7f82c80cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 -- 172.21.15.28:0/2740972637 >> v2:172.21.15.151:3300/0 conn(0x7f82c80cd5d0 msgr2=0x7f82c8099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2740972637 >> v2:172.21.15.151:3300/0 conn(0x7f82c80cd5d0 0x7f82c8099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.387+0000 7f82c6a00640 1 -- 172.21.15.28:0/2740972637 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80c5780 con 0x7f82c815f260 2024-03-21T08:42:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c815f260 0x7f82c80cd090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f82c80730b0 tx=0x7f82b801c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49c72f44446411 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82c5600640 1 -- 172.21.15.28:0/2740972637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b0070 con 0x7f82c815f260 2024-03-21T08:42:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82c5600640 1 -- 172.21.15.28:0/2740972637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b80ab600 con 0x7f82c815f260 2024-03-21T08:42:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82c5600640 1 -- 172.21.15.28:0/2740972637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80ab9d0 con 0x7f82c815f260 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 -- 172.21.15.28:0/2740972637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c815f260 msgr2=0x7f82c80cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c815f260 0x7f82c80cd090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f82c80730b0 tx=0x7f82b801c1d0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 -- 172.21.15.28:0/2740972637 shutdown_connections 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c809a0c0 0x7f82c80cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2740972637 >> v2:172.21.15.151:3300/0 conn(0x7f82c80cd5d0 0x7f82c8099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2740972637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c815f260 0x7f82c80cd090 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 -- 172.21.15.28:0/2740972637 >> 172.21.15.28:0/2740972637 conn(0x7f82c80b34d0 msgr2=0x7f82c80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 -- 172.21.15.28:0/2740972637 shutdown_connections 2024-03-21T08:42:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.391+0000 7f82cdc00640 1 -- 172.21.15.28:0/2740972637 wait complete. 2024-03-21T08:42:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.395+0000 7f82cdc00640 1 Processor -- start 2024-03-21T08:42:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.395+0000 7f82cdc00640 1 -- start start 2024-03-21T08:42:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.395+0000 7f82cdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c80cd5d0 0x7f82c814c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 0x7f82c80b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80c6100 con 0x7f82c809a0c0 2024-03-21T08:42:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80c5f80 con 0x7f82c815f260 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80c6280 con 0x7f82c80cd5d0 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38166/0 (socket says 172.21.15.28:38166) 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 -- 172.21.15.28:0/2665754962 learned_addr learned my addr 172.21.15.28:0/2665754962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6000640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c80cd5d0 0x7f82c814c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c7400640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 0x7f82c80b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 -- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c80cd5d0 msgr2=0x7f82c814c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c80cd5d0 0x7f82c814c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 -- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 msgr2=0x7f82c80b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 0x7f82c80b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 -- 172.21.15.28:0/2665754962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80b5a00 con 0x7f82c809a0c0 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c7400640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 0x7f82c80b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82c6a00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f82b801a880 tx=0x7f82b801aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80ba020 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b80b6620 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80b5ce0 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b0040 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c80b62a0 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c804f410 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.399+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f82b80bd020 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.403+0000 7f82bea00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 0x7f82a8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.403+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f82b8120300 con 0x7f82c809a0c0 2024-03-21T08:42:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.403+0000 7f82c6000640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 0x7f82a8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.403+0000 7f82c6000640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 0x7f82a8069fe0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82c80c4eb0 tx=0x7f82b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.407+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f82b80c4030 con 0x7f82c809a0c0 2024-03-21T08:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.603+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82c8073d20 con 0x7f82c809a0c0 2024-03-21T08:42:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.603+0000 7f82bea00640 1 -- 172.21.15.28:0/2665754962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f82b8016110 con 0x7f82c809a0c0 2024-03-21T08:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:33.074369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:42:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 msgr2=0x7f82a8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 0x7f82a8069fe0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82c80c4eb0 tx=0x7f82b4008040 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 msgr2=0x7f82c8157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f82b801a880 tx=0x7f82b801aa10 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 shutdown_connections 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f82c815f260 0x7f82c80b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f82a8067b20 0x7f82a8069fe0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f82c80cd5d0 0x7f82c814c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 --2- 172.21.15.28:0/2665754962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c809a0c0 0x7f82c8157850 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 >> 172.21.15.28:0/2665754962 conn(0x7f82c80b34d0 msgr2=0x7f82c80ce1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 shutdown_connections 2024-03-21T08:42:40.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:40.607+0000 7f82cdc00640 1 -- 172.21.15.28:0/2665754962 wait complete. 2024-03-21T08:42:41.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-21T08:42:30.068610+0000', 'modified': '2024-03-21T08:42:33.074369+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5517}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5517': {'gid': 5517, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/3938951152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3938951152}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 3938951152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5517, 'qdb_cluster': [5517]} 2024-03-21T08:42:41.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-21T08:42:41.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-21T08:42:41.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.171+0000 7f4f63c00640 1 Processor -- start 2024-03-21T08:42:41.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.171+0000 7f4f63c00640 1 -- start start 2024-03-21T08:42:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f63c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4f5c15e1e0 0x7f4f5c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f63c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c0d0190 0x7f4f5c06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f63c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c06d070 con 0x7f4f5c06cbe0 2024-03-21T08:42:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f63c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0d05f0 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f63c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0b3010 con 0x7f4f5c0d0190 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c0d0190 0x7f4f5c06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4f5c15e1e0 0x7f4f5c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61c00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4f5c06cbe0 0x7f4f5c161c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:39894/0 (socket says 172.21.15.28:39894) 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61c00640 1 -- 172.21.15.28:0/4284895799 learned_addr learned my addr 172.21.15.28:0/4284895799 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61200640 1 -- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c0d0190 msgr2=0x7f4f5c06c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61200640 1 --2- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c0d0190 0x7f4f5c06c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61200640 1 -- 172.21.15.28:0/4284895799 >> v1:172.21.15.151:6789/0 conn(0x7f4f5c06cbe0 legacy=0x7f4f5c161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.175+0000 7f4f61200640 1 -- 172.21.15.28:0/4284895799 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c06d310 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f61200640 1 --2- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0cfc50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f500097c0 tx=0x7f4f5001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af4e3acbb4ccb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f5b400640 1 -- 172.21.15.28:0/4284895799 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500b2070 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f5b400640 1 -- 172.21.15.28:0/4284895799 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500ad640 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f5b400640 1 -- 172.21.15.28:0/4284895799 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500ada10 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 -- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c15e1e0 msgr2=0x7f4f5c0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 --2- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0cfc50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f500097c0 tx=0x7f4f5001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 -- 172.21.15.28:0/4284895799 shutdown_connections 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 --2- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c0d0190 0x7f4f5c06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 --2- 172.21.15.28:0/4284895799 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0cfc50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 -- 172.21.15.28:0/4284895799 >> 172.21.15.28:0/4284895799 conn(0x7f4f5c0d1ab0 msgr2=0x7f4f5c0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 -- 172.21.15.28:0/4284895799 shutdown_connections 2024-03-21T08:42:41.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.179+0000 7f4f63c00640 1 -- 172.21.15.28:0/4284895799 wait complete. 2024-03-21T08:42:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 Processor -- start 2024-03-21T08:42:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- start start 2024-03-21T08:42:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c06cbe0 0x7f4f5c14d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c0d0190 0x7f4f5c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c16b0c0 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c16af40 con 0x7f4f5c0d0190 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c16b240 con 0x7f4f5c06cbe0 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38178/0 (socket says 172.21.15.28:38178) 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 -- 172.21.15.28:0/1949837263 learned_addr learned my addr 172.21.15.28:0/1949837263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f5be00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c0d0190 0x7f4f5c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61200640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c06cbe0 0x7f4f5c14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 -- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c06cbe0 msgr2=0x7f4f5c14d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c06cbe0 0x7f4f5c14d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 -- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c0d0190 msgr2=0x7f4f5c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c0d0190 0x7f4f5c15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 -- 172.21.15.28:0/1949837263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c14d850 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f61c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400ca80 tx=0x7f4f5400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f58c00640 1 -- 172.21.15.28:0/1949837263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54017070 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f58c00640 1 -- 172.21.15.28:0/1949837263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5400ac30 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f58c00640 1 -- 172.21.15.28:0/1949837263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54012400 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c15b810 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.183+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c15bdd0 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f58c00640 1 -- 172.21.15.28:0/1949837263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4f5401a070 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f58c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 0x7f4f3c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 Processor -- start 2024-03-21T08:42:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f58c00640 1 -- 172.21.15.28:0/1949837263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f54085ca0 con 0x7f4f5c15e1e0 2024-03-21T08:42:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 -- start start 2024-03-21T08:42:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f61200640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 0x7f4f3c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82135d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82136240 0x55bd820ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81f39680 con 0x55bd820ebff0 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ec0b00 con 0x55bd82135920 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd81f35720 con 0x55bd82136240 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38192/0 (socket says 172.21.15.28:38192) 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f61c00640 1 -- 172.21.15.28:0/3737978379 learned_addr learned my addr 172.21.15.28:0/3737978379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.187+0000 7f4f61200640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 0x7f4f3c069fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f500af000 tx=0x7f4f50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f5be00640 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82136240 0x55bd820ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61200640 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 -- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82136240 msgr2=0x55bd820ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82136240 0x55bd820ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 -- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 msgr2=0x55bd82135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82135d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 -- 172.21.15.28:0/3737978379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd81f11940 con 0x55bd820ebff0 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4f54055dc0 tx=0x7f4f54055f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1e9fe180ac54e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f4d600640 1 -- 172.21.15.28:0/3737978379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f540524a0 con 0x55bd820ebff0 2024-03-21T08:42:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f4d600640 1 -- 172.21.15.28:0/3737978379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5405e270 con 0x55bd820ebff0 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f4d600640 1 -- 172.21.15.28:0/3737978379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54052640 con 0x55bd820ebff0 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- 172.21.15.28:0/3737978379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 msgr2=0x55bd820ec3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4f54055dc0 tx=0x7f4f54055f70 comp rx=0 tx=0).stop 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- 172.21.15.28:0/3737978379 shutdown_connections 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd820ec3d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82136240 0x55bd820ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- 172.21.15.28:0/3737978379 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- 172.21.15.28:0/3737978379 >> 172.21.15.28:0/3737978379 conn(0x55bd820e7fa0 msgr2=0x55bd820ea390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- 172.21.15.28:0/3737978379 shutdown_connections 2024-03-21T08:42:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- 172.21.15.28:0/3737978379 wait complete. 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- start start 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 Processor -- start 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd820ebff0 0x55bd8211ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd8211b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81f38990 con 0x55bd82136240 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ef3ae0 con 0x55bd820ebff0 2024-03-21T08:42:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ed1750 con 0x55bd82135920 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f671ac000 10 client.0 _ll_register_callbacks cb 0x55bd820d6ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38200/0 (socket says 172.21.15.28:38200) 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61c00640 1 -- 172.21.15.28:0/858868339 learned_addr learned my addr 172.21.15.28:0/858868339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f61200640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd820ebff0 0x55bd8211ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.191+0000 7f4f5be00640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd8211b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 -- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 msgr2=0x55bd8211b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd8211b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 -- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd820ebff0 msgr2=0x55bd8211ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd820ebff0 0x55bd8211ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 -- 172.21.15.28:0/858868339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd81f7ed80 con 0x55bd82136240 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61c00640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4f540540e0 tx=0x7f4f54001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb74702b453e1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:42:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54055280 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f54055420 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f5405e270 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5405c0e0 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 10 client.5730 fetch_fsmap learned FSMap version 18 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55bd81ec0d10 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5405e270 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f5405c280 con 0x55bd82136240 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 10 client.5730 fetch_fsmap finished waiting for FSMap version 18 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 10 client.5730 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 20 client.5730 populate_metadata read hostname 'smithi028' 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8211c060 0x55bd82147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 4 client.5730 mds_command: new command op to 5517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 --> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd81ed4b60 con 0x55bd8211c060 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61200640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8211c060 0x55bd82147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f61200640 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8211c060 0x55bd82147ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61a5b8a7076681ae server_cookie=449011018a5c4ac in_seq=0 out_seq=0 2024-03-21T08:42:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 10 client.5730 ms_handle_connect on v2:172.21.15.28:6832/3938951152 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 1 -- 172.21.15.28:0/858868339 <== mds.0 v2:172.21.15.28:6832/3938951152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bd81ed4b60 con 0x55bd8211c060 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 10 client.5730 handle_command_reply: tid=0 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 client.5730 shutdown 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 msgr2=0x55bd8211b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4f540540e0 tx=0x7f4f54001260 comp rx=0 tx=0).stop 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 shutdown_connections 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8211c060 0x55bd82147ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd8211b7c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd8211b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f42000640 0 client.5730 ms_handle_reset on v2:172.21.15.28:6832/3938951152 2024-03-21T08:42:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 --2- 172.21.15.28:0/858868339 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd820ebff0 0x55bd8211ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 >> 172.21.15.28:0/858868339 conn(0x55bd820e7fa0 msgr2=0x55bd820eac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 shutdown_connections 2024-03-21T08:42:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.195+0000 7f4f671ac000 1 -- 172.21.15.28:0/858868339 wait complete. 2024-03-21T08:42:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.199+0000 7f4f671ac000 20 client.5730 trim_cache size 0 max 16384 2024-03-21T08:42:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 Processor -- start 2024-03-21T08:42:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- start start 2024-03-21T08:42:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd82134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82136240 0x55bd82197170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c14e620 con 0x55bd820ebff0 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c14ec20 con 0x55bd82136240 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ee1a80 con 0x55bd82135920 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38216/0 (socket says 172.21.15.28:38216) 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 -- 172.21.15.28:0/1078031174 learned_addr learned my addr 172.21.15.28:0/1078031174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f5be00640 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd82134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61c00640 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82136240 0x55bd82197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 -- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 msgr2=0x55bd82134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd82134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 -- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82136240 msgr2=0x55bd82197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82136240 0x55bd82197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 -- 172.21.15.28:0/1078031174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd81f06540 con 0x55bd820ebff0 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f61200640 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b0040 tx=0x7f4f50004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d4bfd48bac9fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f4cc00640 1 -- 172.21.15.28:0/1078031174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500b2040 con 0x55bd820ebff0 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f4cc00640 1 -- 172.21.15.28:0/1078031174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500d2070 con 0x55bd820ebff0 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f4cc00640 1 -- 172.21.15.28:0/1078031174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500ada80 con 0x55bd820ebff0 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- 172.21.15.28:0/1078031174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 msgr2=0x55bd82134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f500b0040 tx=0x7f4f50004740 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- 172.21.15.28:0/1078031174 shutdown_connections 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82136240 0x55bd82197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd82135920 0x55bd82134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 --2- 172.21.15.28:0/1078031174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd820ebff0 0x55bd82134500 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- 172.21.15.28:0/1078031174 >> 172.21.15.28:0/1078031174 conn(0x55bd820e7fa0 msgr2=0x55bd82142f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- 172.21.15.28:0/1078031174 shutdown_connections 2024-03-21T08:42:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- 172.21.15.28:0/1078031174 wait complete. 2024-03-21T08:42:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.215+0000 7f4f671ac000 1 -- start start 2024-03-21T08:42:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 Processor -- start 2024-03-21T08:42:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd820ebff0 0x55bd820eb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd82149430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ed1750 con 0x55bd82136240 2024-03-21T08:42:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd81ff6700 con 0x55bd82135920 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd81f38990 con 0x55bd820ebff0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 10 client.0 _ll_register_callbacks cb 0x7f4f500b0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd82149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35280/0 (socket says 172.21.15.28:35280) 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f5be00640 1 -- 172.21.15.28:0/2646730854 learned_addr learned my addr 172.21.15.28:0/2646730854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61200640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd820ebff0 0x55bd820eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 -- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd820ebff0 msgr2=0x55bd820eb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd820ebff0 0x55bd820eb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 -- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 msgr2=0x55bd82148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 -- 172.21.15.28:0/2646730854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd81ecd4b0 con 0x55bd82136240 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61c00640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd82149430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4f5405e1d0 tx=0x7f4f54052150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0e8877ac0de74 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54084170 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f54013de0 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f500d2070 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f54051ed0 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 10 client.5736 fetch_fsmap learned FSMap version 18 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55bd81ee02f0 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f500d2070 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f5405e270 con 0x55bd82136240 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 10 client.5736 fetch_fsmap finished waiting for FSMap version 18 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 10 client.5736 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 20 client.5736 populate_metadata read hostname 'smithi028' 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8219a7d0 0x55bd8219abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 4 client.5736 mds_command: new command op to 5517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T08:42:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 --> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bd81f46bc0 con 0x55bd8219a7d0 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61200640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8219a7d0 0x55bd8219abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f61200640 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8219a7d0 0x55bd8219abb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a283f75881dd1b9 server_cookie=b54648835d6f07a5 in_seq=0 out_seq=0 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.219+0000 7f4f42000640 10 client.5736 ms_handle_connect on v2:172.21.15.28:6832/3938951152 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f42000640 1 -- 172.21.15.28:0/2646730854 <== mds.0 v2:172.21.15.28:6832/3938951152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55bd81f46bc0 con 0x55bd8219a7d0 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f42000640 10 client.5736 handle_command_reply: tid=0 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 client.5736 shutdown 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 msgr2=0x55bd82149430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd82149430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4f5405e1d0 tx=0x7f4f54052150 comp rx=0 tx=0).stop 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 shutdown_connections 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:6832/3938951152,v1:172.21.15.28:6834/3938951152] conn(0x55bd8219a7d0 0x55bd8219abb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd82136240 0x55bd82149430 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55bd82135920 0x55bd82148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 --2- 172.21.15.28:0/2646730854 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55bd820ebff0 0x55bd820eb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 >> 172.21.15.28:0/2646730854 conn(0x55bd820e7fa0 msgr2=0x55bd82142680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f42000640 0 client.5736 ms_handle_reset on v2:172.21.15.28:6832/3938951152 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 shutdown_connections 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 1 -- 172.21.15.28:0/2646730854 wait complete. 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f671ac000 20 client.5736 trim_cache size 0 max 16384 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "51dba44a-cba1-4b57-a836-26a407968834", 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5517, 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-21T08:42:41.229 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 10.544631031, 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 18, 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 39, 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 39, 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 18.400433202999999 2024-03-21T08:42:41.230 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 msgr2=0x7f4f3c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 0x7f4f3c069fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f500af000 tx=0x7f4f50008040 comp rx=0 tx=0).stop 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 msgr2=0x7f4f5c0b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400ca80 tx=0x7f4f5400cf40 comp rx=0 tx=0).stop 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 shutdown_connections 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c15e1e0 0x7f4f5c0b3580 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4f5c0d0190 0x7f4f5c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4f3c067b20 0x7f4f3c069fe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 --2- 172.21.15.28:0/1949837263 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4f5c06cbe0 0x7f4f5c14d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 >> 172.21.15.28:0/1949837263 conn(0x7f4f5c0d1ab0 msgr2=0x7f4f5c0ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 shutdown_connections 2024-03-21T08:42:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.223+0000 7f4f63c00640 1 -- 172.21.15.28:0/1949837263 wait complete. 2024-03-21T08:42:41.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:42:41.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.383+0000 7f1418800640 1 Processor -- start 2024-03-21T08:42:41.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.383+0000 7f1418800640 1 -- start start 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1418800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1410099b80 0x7f1410099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1418800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f14100c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1418800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100cdb50 con 0x7f141015f260 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1418800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141009a520 con 0x7f1410099b80 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1418800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14100c5530 con 0x7f14100c5f80 2024-03-21T08:42:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1410099b80 0x7f1410099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1410099b80 0x7f1410099fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38238/0 (socket says 172.21.15.28:38238) 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 -- 172.21.15.28:0/3951385605 learned_addr learned my addr 172.21.15.28:0/3951385605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1416800640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f14100c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 -- 172.21.15.28:0/3951385605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 msgr2=0x7f14100c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f14100c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 -- 172.21.15.28:0/3951385605 >> v1:172.21.15.151:6789/0 conn(0x7f141015f260 legacy=0x7f14100cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 -- 172.21.15.28:0/3951385605 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14100cde30 con 0x7f1410099b80 2024-03-21T08:42:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.387+0000 7f1415400640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410099fe0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1400009870 tx=0x7f140001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe851217aaefdef server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1414a00640 1 -- 172.21.15.28:0/3951385605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000aa600 con 0x7f1410099b80 2024-03-21T08:42:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1414a00640 1 -- 172.21.15.28:0/3951385605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14000af070 con 0x7f1410099b80 2024-03-21T08:42:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1414a00640 1 -- 172.21.15.28:0/3951385605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000aa9c0 con 0x7f1410099b80 2024-03-21T08:42:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 -- 172.21.15.28:0/3951385605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 msgr2=0x7f1410099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410099fe0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1400009870 tx=0x7f140001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 -- 172.21.15.28:0/3951385605 shutdown_connections 2024-03-21T08:42:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f14100c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 --2- 172.21.15.28:0/3951385605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410099fe0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 -- 172.21.15.28:0/3951385605 >> 172.21.15.28:0/3951385605 conn(0x7f14100b34d0 msgr2=0x7f14100b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 -- 172.21.15.28:0/3951385605 shutdown_connections 2024-03-21T08:42:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 -- 172.21.15.28:0/3951385605 wait complete. 2024-03-21T08:42:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.391+0000 7f1418800640 1 Processor -- start 2024-03-21T08:42:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- start start 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f1410157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 0x7f1410163d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14100cd7a0 con 0x7f1410099b80 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141015f640 con 0x7f141015f260 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1410164fc0 con 0x7f14100c5f80 2024-03-21T08:42:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38246/0 (socket says 172.21.15.28:38246) 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 -- 172.21.15.28:0/2347534940 learned_addr learned my addr 172.21.15.28:0/2347534940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415400640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f1410157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1416800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 0x7f1410163d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 -- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 msgr2=0x7f1410157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f1410157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 -- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 msgr2=0x7f1410163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 0x7f1410163d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 -- 172.21.15.28:0/2347534940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14101652a0 con 0x7f1410099b80 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1416800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 0x7f1410163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1415e00640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f140400c500 tx=0x7f140400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404016020 con 0x7f1410099b80 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1404005220 con 0x7f1410099b80 2024-03-21T08:42:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1410165580 con 0x7f1410099b80 2024-03-21T08:42:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1404007030 con 0x7f1410099b80 2024-03-21T08:42:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.395+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14100cb4c0 con 0x7f1410099b80 2024-03-21T08:42:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14100cb7b0 con 0x7f1410099b80 2024-03-21T08:42:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f14040071f0 con 0x7f1410099b80 2024-03-21T08:42:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f140a000640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 0x7f13f4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1404085ab0 con 0x7f1410099b80 2024-03-21T08:42:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f1415400640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 0x7f13f4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.399+0000 7f1415400640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 0x7f13f4069fe0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f14000ae000 tx=0x7f14000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.403+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1404020020 con 0x7f1410099b80 2024-03-21T08:42:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f141004f410 con 0x7f1410099b80 2024-03-21T08:42:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f140a000640 1 -- 172.21.15.28:0/2347534940 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f14040562a0 con 0x7f1410099b80 2024-03-21T08:42:41.609 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-21T08:42:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 msgr2=0x7f13f4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 0x7f13f4069fe0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f14000ae000 tx=0x7f14000a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 msgr2=0x7f1410156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f140400c500 tx=0x7f140400c9c0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 shutdown_connections 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f141015f260 0x7f1410163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f13f4067b20 0x7f13f4069fe0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14100c5f80 0x7f1410157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 --2- 172.21.15.28:0/2347534940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1410099b80 0x7f1410156bc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.607+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 >> 172.21.15.28:0/2347534940 conn(0x7f14100b34d0 msgr2=0x7f14100cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.611+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 shutdown_connections 2024-03-21T08:42:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:41.611+0000 7f1418800640 1 -- 172.21.15.28:0/2347534940 wait complete. 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:42:41.631 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:42:41.632 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:41.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.632 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.641 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42308sec preferred_lft 42308sec 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.642 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:42:41.643 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:41.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.645 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:42:41.704 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:42:41.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.707 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:42:41.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:42:41.762 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:42:41.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.854 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:42:41.863 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:42:41.863 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:42:41.864 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:41.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:41.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:42:41.927 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:42:41.927 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:42:41.927 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:41.979 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:41.979 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:42:41.979 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:42.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:42:42.091 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:42:42.144 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:42:42.144 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:42:42.144 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:42:42.145 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:42:42.146 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:42.147 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:42.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:42.148 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:42:42.204 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:42:42.204 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:42:42.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:42:42.277 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:42:42.330 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:42:42.331 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:42.332 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:42.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:42.333 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:42:42.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:42:42.359+0000 7f79c0923580 -1 init, newargv = 0x55806cfb7670 newargc=15 2024-03-21T08:42:42.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[37535]: starting ceph client 2024-03-21T08:42:42.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[37535]: starting fuse 2024-03-21T08:42:42.373 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:42:42.373 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:42:42.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:42:42.373 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:42:42.373 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:42:42.373 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:42:42.374 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:42:42.375 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:42:42.503 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.37535.asok 2024-03-21T08:42:42.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:42:42.514 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.37535.asok 2024-03-21T08:42:42.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:42.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.37535.asok status 2024-03-21T08:42:42.742 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "37535", 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5751, 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:42:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5751 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1517439068 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1517439068 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.5751 192.168.0.1:0/1517439068", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1517439068", 2024-03-21T08:42:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 18, 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 39, 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:42:42.745 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:42:42.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:42.764 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:42:42.764 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:42.773 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:42:42.774 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:42.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:42.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:42.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-21T08:42:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 Processor -- start 2024-03-21T08:42:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 -- start start 2024-03-21T08:42:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01200cf9b0 0x7f01200ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200d0040 con 0x7f012015ba60 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01200cfe00 con 0x7f01200cf9b0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01200d0320 con 0x7f01200b0f60 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01200cf9b0 0x7f01200ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01200cf9b0 0x7f01200ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38258/0 (socket says 172.21.15.28:38258) 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 -- 172.21.15.28:0/1364585648 learned_addr learned my addr 172.21.15.28:0/1364585648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f0115400640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 -- 172.21.15.28:0/1364585648 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 msgr2=0x7f01200d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200d10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 -- 172.21.15.28:0/1364585648 >> v1:172.21.15.151:6789/0 conn(0x7f012015ba60 legacy=0x7f01200b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.023+0000 7f011e800640 1 -- 172.21.15.28:0/1364585648 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01200d0600 con 0x7f01200cf9b0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011e800640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cf9b0 0x7f01200ce820 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0118009870 tx=0x7f011801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8421ce7b7f2227 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011d400640 1 -- 172.21.15.28:0/1364585648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180b5070 con 0x7f01200cf9b0 2024-03-21T08:42:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011d400640 1 -- 172.21.15.28:0/1364585648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01180a8a50 con 0x7f01200cf9b0 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011d400640 1 -- 172.21.15.28:0/1364585648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180a8e20 con 0x7f01200cf9b0 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- 172.21.15.28:0/1364585648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cf9b0 msgr2=0x7f01200ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cf9b0 0x7f01200ce820 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0118009870 tx=0x7f011801ed50 comp rx=0 tx=0).stop 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- 172.21.15.28:0/1364585648 shutdown_connections 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01200cf9b0 0x7f01200ce820 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 --2- 172.21.15.28:0/1364585648 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- 172.21.15.28:0/1364585648 >> 172.21.15.28:0/1364585648 conn(0x7f012006fa90 msgr2=0x7f012006fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- 172.21.15.28:0/1364585648 shutdown_connections 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- 172.21.15.28:0/1364585648 wait complete. 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 Processor -- start 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.027+0000 7f011f200640 1 -- start start 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 0x7f012014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012015be40 con 0x7f012015ba60 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200d0bc0 con 0x7f01200cf9b0 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012015bfc0 con 0x7f01200b0f60 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38266/0 (socket says 172.21.15.28:38266) 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 -- 172.21.15.28:0/4145762167 learned_addr learned my addr 172.21.15.28:0/4145762167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f0115400640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 0x7f012014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011de00640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 -- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 msgr2=0x7f01200c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 -- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 msgr2=0x7f012014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 0x7f012014ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 -- 172.21.15.28:0/4145762167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01201e6810 con 0x7f012015ba60 2024-03-21T08:42:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f0115400640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 0x7f012014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011de00640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011e800640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01180b3040 tx=0x7f011801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180b5070 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01180b0d80 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01200cb630 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180bb630 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01200cbbf0 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.031+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01201e69e0 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.035+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f01180c2060 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.035+0000 7f0116a00640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 0x7f00fc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.035+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f01181235b0 con 0x7f012015ba60 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.035+0000 7f011de00640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 0x7f00fc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.035+0000 7f011de00640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 0x7f00fc069fe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0110001650 tx=0x7f0110009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.043+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f01180188b0 con 0x7f012015ba60 2024-03-21T08:42:43.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.335+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) v1 -- 0x7f012004f410 con 0x7f012015ba60 2024-03-21T08:42:43.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.803+0000 7f0116a00640 1 -- 172.21.15.28:0/4145762167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v40) v1 ==== 124+0+0 (secure 0 0 0) 0x7f01180f3d90 con 0x7f012015ba60 2024-03-21T08:42:43.806 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'data_new' created 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 msgr2=0x7f00fc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 0x7f00fc069fe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0110001650 tx=0x7f0110009040 comp rx=0 tx=0).stop 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 msgr2=0x7f012014cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01180b3040 tx=0x7f011801baf0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 shutdown_connections 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012015ba60 0x7f012014cfa0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f01200cf9b0 0x7f012014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f00fc067b20 0x7f00fc069fe0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 --2- 172.21.15.28:0/4145762167 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f01200b0f60 0x7f01200c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 >> 172.21.15.28:0/4145762167 conn(0x7f012006fa90 msgr2=0x7f01200b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 shutdown_connections 2024-03-21T08:42:43.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.807+0000 7f011f200640 1 -- 172.21.15.28:0/4145762167 wait complete. 2024-03-21T08:42:43.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-21T08:42:43.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 Processor -- start 2024-03-21T08:42:43.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 -- start start 2024-03-21T08:42:43.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef7c15bd20 0x7fef7c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 0x7fef7c08afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c15fba0 con 0x7fef7c0b0e80 2024-03-21T08:42:43.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c15fe80 con 0x7fef7c15bd20 2024-03-21T08:42:43.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef84000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c160160 con 0x7fef7c0cebd0 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef81600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef7c15bd20 0x7fef7c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef80c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fef7c0b0e80 0x7fef7c0ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41826/0 (socket says 172.21.15.28:41826) 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef80c00640 1 -- 172.21.15.28:0/894003293 learned_addr learned my addr 172.21.15.28:0/894003293 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef82000640 1 --2- 172.21.15.28:0/894003293 >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 0x7fef7c08afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef81600640 1 -- 172.21.15.28:0/894003293 >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 msgr2=0x7fef7c08afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef81600640 1 --2- 172.21.15.28:0/894003293 >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 0x7fef7c08afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef81600640 1 -- 172.21.15.28:0/894003293 >> v1:172.21.15.28:6789/0 conn(0x7fef7c0b0e80 legacy=0x7fef7c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef81600640 1 -- 172.21.15.28:0/894003293 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c160440 con 0x7fef7c15bd20 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef82000640 1 --2- 172.21.15.28:0/894003293 >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 0x7fef7c08afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.979+0000 7fef7be00640 1 -- 172.21.15.28:0/894003293 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786361671 0 0) 0x7fef7c15fba0 con 0x7fef7c0b0e80 2024-03-21T08:42:43.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef81600640 1 --2- 172.21.15.28:0/894003293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c0b2c40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0b1960 tx=0x7fef7001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edfcf0f93a882f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:43.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef7be00640 1 -- 172.21.15.28:0/894003293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700b0070 con 0x7fef7c15bd20 2024-03-21T08:42:43.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef7be00640 1 -- 172.21.15.28:0/894003293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef700ab600 con 0x7fef7c15bd20 2024-03-21T08:42:43.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef7be00640 1 -- 172.21.15.28:0/894003293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700ab9d0 con 0x7fef7c15bd20 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 -- 172.21.15.28:0/894003293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 msgr2=0x7fef7c0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 --2- 172.21.15.28:0/894003293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c0b2c40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0b1960 tx=0x7fef7001c480 comp rx=0 tx=0).stop 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 -- 172.21.15.28:0/894003293 shutdown_connections 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 --2- 172.21.15.28:0/894003293 >> v2:172.21.15.151:3300/0 conn(0x7fef7c0cebd0 0x7fef7c08afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 --2- 172.21.15.28:0/894003293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c0b2c40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 -- 172.21.15.28:0/894003293 >> 172.21.15.28:0/894003293 conn(0x7fef7c0d1ab0 msgr2=0x7fef7c0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 -- 172.21.15.28:0/894003293 shutdown_connections 2024-03-21T08:42:43.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.983+0000 7fef84000640 1 -- 172.21.15.28:0/894003293 wait complete. 2024-03-21T08:42:43.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.987+0000 7fef84000640 1 Processor -- start 2024-03-21T08:42:43.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.987+0000 7fef84000640 1 -- start start 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.987+0000 7fef84000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 0x7fef7c06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef7c0cebd0 0x7fef7c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c08bd10 con 0x7fef7c15bd20 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c15c310 con 0x7fef7c0b0e80 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c08be90 con 0x7fef7c0cebd0 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38290/0 (socket says 172.21.15.28:38290) 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 -- 172.21.15.28:0/408560401 learned_addr learned my addr 172.21.15.28:0/408560401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef81600640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 0x7fef7c06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef80c00640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef7c0cebd0 0x7fef7c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 -- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef7c0cebd0 msgr2=0x7fef7c15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef7c0cebd0 0x7fef7c15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 -- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 msgr2=0x7fef7c06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 0x7fef7c06c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 -- 172.21.15.28:0/408560401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c06c950 con 0x7fef7c15bd20 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef81600640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 0x7fef7c06c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef82000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fef7400cef0 tx=0x7fef7400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7401a070 con 0x7fef7c15bd20 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7400d860 con 0x7fef7c15bd20 2024-03-21T08:42:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7400dc50 con 0x7fef7c15bd20 2024-03-21T08:42:43.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c06cc30 con 0x7fef7c15bd20 2024-03-21T08:42:43.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c06d1f0 con 0x7fef7c15bd20 2024-03-21T08:42:43.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c15f2b0 con 0x7fef7c15bd20 2024-03-21T08:42:43.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.991+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fef7401d070 con 0x7fef7c15bd20 2024-03-21T08:42:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.995+0000 7fef79600640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 0x7fef5c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.995+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6464+0+0 (secure 0 0 0) 0x7fef74086ad0 con 0x7fef7c15bd20 2024-03-21T08:42:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.995+0000 7fef81600640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 0x7fef5c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:43.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.995+0000 7fef81600640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 0x7fef5c069fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef700ad000 tx=0x7fef700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:44.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:43.999+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fef740572c0 con 0x7fef7c15bd20 2024-03-21T08:42:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.195+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) v1 -- 0x7fef7c0b7790 con 0x7fef7c15bd20 2024-03-21T08:42:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.819+0000 7fef79600640 1 -- 172.21.15.28:0/408560401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v19) v1 ==== 131+0+0 (secure 0 0 0) 0x7fef74056bc0 con 0x7fef7c15bd20 2024-03-21T08:42:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 7 to fsmap 2024-03-21T08:42:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.823+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 msgr2=0x7fef5c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.823+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 0x7fef5c069fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef700ad000 tx=0x7fef700a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.823+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 msgr2=0x7fef7c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.823+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fef7400cef0 tx=0x7fef7400a710 comp rx=0 tx=0).stop 2024-03-21T08:42:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 shutdown_connections 2024-03-21T08:42:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef7c15bd20 0x7fef7c15d0e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fef5c067b20 0x7fef5c069fe0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fef7c0cebd0 0x7fef7c15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 --2- 172.21.15.28:0/408560401 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fef7c0b0e80 0x7fef7c06c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.827+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 >> 172.21.15.28:0/408560401 conn(0x7fef7c0d1ab0 msgr2=0x7fef7c0cc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.831+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 shutdown_connections 2024-03-21T08:42:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.831+0000 7fef84000640 1 -- 172.21.15.28:0/408560401 wait complete. 2024-03-21T08:42:44.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:42:45.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 Processor -- start 2024-03-21T08:42:45.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 -- start start 2024-03-21T08:42:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f26a815c680 0x7f26a80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26a806b170 0x7f26a806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a80ceaa0 con 0x7f26a80cfaa0 2024-03-21T08:42:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a806bb00 con 0x7f26a806b170 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:44.999+0000 7f26ae600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a80ced80 con 0x7f26a815c680 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26a806b170 0x7f26a806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26a806b170 0x7f26a806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38306/0 (socket says 172.21.15.28:38306) 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 -- 172.21.15.28:0/1223239184 learned_addr learned my addr 172.21.15.28:0/1223239184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7400640 1 --2- 172.21.15.28:0/1223239184 >> v2:172.21.15.151:3300/0 conn(0x7f26a815c680 0x7f26a80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 -- 172.21.15.28:0/1223239184 >> v2:172.21.15.151:3300/0 conn(0x7f26a815c680 msgr2=0x7f26a80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 --2- 172.21.15.28:0/1223239184 >> v2:172.21.15.151:3300/0 conn(0x7f26a815c680 0x7f26a80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 -- 172.21.15.28:0/1223239184 >> v1:172.21.15.151:6789/0 conn(0x7f26a80cfaa0 legacy=0x7f26a815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 -- 172.21.15.28:0/1223239184 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80cf060 con 0x7f26a806b170 2024-03-21T08:42:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a7e00640 1 --2- 172.21.15.28:0/1223239184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a806b5c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f26a809a5b0 tx=0x7f26a001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7235cce20b669cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a6a00640 1 -- 172.21.15.28:0/1223239184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a00a5620 con 0x7f26a806b170 2024-03-21T08:42:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a6a00640 1 -- 172.21.15.28:0/1223239184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a00a57c0 con 0x7f26a806b170 2024-03-21T08:42:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.003+0000 7f26a6a00640 1 -- 172.21.15.28:0/1223239184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a00a5b90 con 0x7f26a806b170 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- 172.21.15.28:0/1223239184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 msgr2=0x7f26a806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- 172.21.15.28:0/1223239184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a806b5c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f26a809a5b0 tx=0x7f26a001c960 comp rx=0 tx=0).stop 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- 172.21.15.28:0/1223239184 shutdown_connections 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- 172.21.15.28:0/1223239184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a806b5c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- 172.21.15.28:0/1223239184 >> v2:172.21.15.151:3300/0 conn(0x7f26a815c680 0x7f26a80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- 172.21.15.28:0/1223239184 >> 172.21.15.28:0/1223239184 conn(0x7f26a8099a30 msgr2=0x7f26a8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- 172.21.15.28:0/1223239184 shutdown_connections 2024-03-21T08:42:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- 172.21.15.28:0/1223239184 wait complete. 2024-03-21T08:42:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 Processor -- start 2024-03-21T08:42:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- start start 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26a80cfaa0 0x7f26a814fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26a815c680 0x7f26a815abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a80cd500 con 0x7f26a806b170 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a80cd380 con 0x7f26a815c680 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26ae600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a8168aa0 con 0x7f26a80cfaa0 2024-03-21T08:42:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38312/0 (socket says 172.21.15.28:38312) 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 -- 172.21.15.28:0/4182142016 learned_addr learned my addr 172.21.15.28:0/4182142016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7e00640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26a815c680 0x7f26a815abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f269fe00640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26a80cfaa0 0x7f26a814fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 -- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26a80cfaa0 msgr2=0x7f26a814fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26a80cfaa0 0x7f26a814fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 -- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26a815c680 msgr2=0x7f26a815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26a815c680 0x7f26a815abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.007+0000 7f26a7400640 1 -- 172.21.15.28:0/4182142016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80b4ed0 con 0x7f26a806b170 2024-03-21T08:42:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f26a7400640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f269800d800 tx=0x7f269800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269800b6c0 con 0x7f26a806b170 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2698019070 con 0x7f26a806b170 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a80b51b0 con 0x7f26a806b170 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269800ba60 con 0x7f26a806b170 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a80b5770 con 0x7f26a806b170 2024-03-21T08:42:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a804f410 con 0x7f26a806b170 2024-03-21T08:42:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f269800bc00 con 0x7f26a806b170 2024-03-21T08:42:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 0x7f268406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2698086680 con 0x7f26a806b170 2024-03-21T08:42:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.011+0000 7f269fe00640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 0x7f268406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.015+0000 7f269fe00640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 0x7f268406a010 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f26a81f9e90 tx=0x7f2690009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.015+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2698056cf0 con 0x7f26a806b170 2024-03-21T08:42:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.215+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a80d1440 con 0x7f26a806b170 2024-03-21T08:42:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f269f400640 1 -- 172.21.15.28:0/4182142016 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f26980565f0 con 0x7f26a806b170 2024-03-21T08:42:45.219 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:45.220 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:44.201277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:42:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 msgr2=0x7f268406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 0x7f268406a010 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f26a81f9e90 tx=0x7f2690009040 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 msgr2=0x7f26a80b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f269800d800 tx=0x7f269800dcc0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 shutdown_connections 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26a815c680 0x7f26a815abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2684067b50 0x7f268406a010 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f26a80cfaa0 0x7f26a814fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 --2- 172.21.15.28:0/4182142016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a806b170 0x7f26a80b4530 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 >> 172.21.15.28:0/4182142016 conn(0x7f26a8099a30 msgr2=0x7f26a815e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 shutdown_connections 2024-03-21T08:42:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.219+0000 7f26ae600640 1 -- 172.21.15.28:0/4182142016 wait complete. 2024-03-21T08:42:45.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 Processor -- start 2024-03-21T08:42:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 -- start start 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b840b2e30 0x7f5b840ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406b310 con 0x7f5b840cfa80 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8406b5f0 con 0x7f5b840cc880 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b8a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8415e1c0 con 0x7f5b840b2e30 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b83e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b840b2e30 0x7f5b840ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b88c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5b840cfa80 0x7f5b840cfe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41836/0 (socket says 172.21.15.28:41836) 2024-03-21T08:42:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.379+0000 7f5b88c00640 1 -- 172.21.15.28:0/3078317698 learned_addr learned my addr 172.21.15.28:0/3078317698 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83e00640 1 -- 172.21.15.28:0/3078317698 >> v1:172.21.15.151:6789/0 conn(0x7f5b840cc880 legacy=0x7f5b840cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83e00640 1 -- 172.21.15.28:0/3078317698 >> v1:172.21.15.28:6789/0 conn(0x7f5b840cfa80 legacy=0x7f5b840cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83e00640 1 -- 172.21.15.28:0/3078317698 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8406b8d0 con 0x7f5b840b2e30 2024-03-21T08:42:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83400640 1 -- 172.21.15.28:0/3078317698 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132421064 0 0) 0x7f5b8406b310 con 0x7f5b840cfa80 2024-03-21T08:42:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83e00640 1 --2- 172.21.15.28:0/3078317698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840b2e30 0x7f5b840ccf60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5b700097c0 tx=0x7f5b7001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca43ae9599ea57ad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83400640 1 -- 172.21.15.28:0/3078317698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b700aa5f0 con 0x7f5b840b2e30 2024-03-21T08:42:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83400640 1 -- 172.21.15.28:0/3078317698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b700af070 con 0x7f5b840b2e30 2024-03-21T08:42:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b83400640 1 -- 172.21.15.28:0/3078317698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b700aa9b0 con 0x7f5b840b2e30 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b8a000640 1 -- 172.21.15.28:0/3078317698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840b2e30 msgr2=0x7f5b840ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b8a000640 1 --2- 172.21.15.28:0/3078317698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840b2e30 0x7f5b840ccf60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5b700097c0 tx=0x7f5b7001c470 comp rx=0 tx=0).stop 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b8a000640 1 -- 172.21.15.28:0/3078317698 shutdown_connections 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b8a000640 1 --2- 172.21.15.28:0/3078317698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840b2e30 0x7f5b840ccf60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.383+0000 7f5b8a000640 1 -- 172.21.15.28:0/3078317698 >> 172.21.15.28:0/3078317698 conn(0x7f5b840b2280 msgr2=0x7f5b840b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.387+0000 7f5b8a000640 1 -- 172.21.15.28:0/3078317698 shutdown_connections 2024-03-21T08:42:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.387+0000 7f5b8a000640 1 -- 172.21.15.28:0/3078317698 wait complete. 2024-03-21T08:42:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.387+0000 7f5b8a000640 1 Processor -- start 2024-03-21T08:42:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.387+0000 7f5b8a000640 1 -- start start 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b840cc880 0x7f5b8414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 0x7f5b8414ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8416c2f0 con 0x7f5b840cfa80 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840cdfc0 con 0x7f5b840b2e30 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8416c470 con 0x7f5b840cc880 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 0x7f5b8414ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b88c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b88c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35332/0 (socket says 172.21.15.28:35332) 2024-03-21T08:42:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b88c00640 1 -- 172.21.15.28:0/1512482283 learned_addr learned my addr 172.21.15.28:0/1512482283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b83e00640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b840cc880 0x7f5b8414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 -- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b840cc880 msgr2=0x7f5b8414c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b840cc880 0x7f5b8414c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 -- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 msgr2=0x7f5b840c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 -- 172.21.15.28:0/1512482283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8414fc70 con 0x7f5b840cfa80 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b88c00640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b89600640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 0x7f5b8414ccf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5b840cccb0 tx=0x7f5b7c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c003b80 con 0x7f5b840cfa80 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c003d20 con 0x7f5b840cfa80 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c005310 con 0x7f5b840cfa80 2024-03-21T08:42:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8414ff50 con 0x7f5b840cfa80 2024-03-21T08:42:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b840c6f00 con 0x7f5b840cfa80 2024-03-21T08:42:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.391+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8406d210 con 0x7f5b840cfa80 2024-03-21T08:42:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.395+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5b7c0056c0 con 0x7f5b840cfa80 2024-03-21T08:42:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.395+0000 7f5b80c00640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 0x7f5b64069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.395+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b7c086140 con 0x7f5b840cfa80 2024-03-21T08:42:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.395+0000 7f5b88c00640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 0x7f5b64069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.395+0000 7f5b88c00640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 0x7f5b64069fe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b78007b20 tx=0x7f5b78007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.399+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5b7c0561f0 con 0x7f5b840cfa80 2024-03-21T08:42:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.559+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b84150ff0 con 0x7f5b840cfa80 2024-03-21T08:42:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b80c00640 1 -- 172.21.15.28:0/1512482283 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18779 (secure 0 0 0) 0x7f5b7c05a290 con 0x7f5b840cfa80 2024-03-21T08:42:45.565 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:45.565 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:44.200971+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 msgr2=0x7f5b64069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 0x7f5b64069fe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b78007b20 tx=0x7f5b78007a70 comp rx=0 tx=0).stop 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 msgr2=0x7f5b8414ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 0x7f5b8414ccf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5b840cccb0 tx=0x7f5b7c00cb90 comp rx=0 tx=0).stop 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 shutdown_connections 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b840cfa80 0x7f5b8414ccf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5b64067b20 0x7f5b64069fe0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5b840cc880 0x7f5b8414c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 --2- 172.21.15.28:0/1512482283 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5b840b2e30 0x7f5b840c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 >> 172.21.15.28:0/1512482283 conn(0x7f5b840b2280 msgr2=0x7f5b840cb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:45.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 shutdown_connections 2024-03-21T08:42:45.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.563+0000 7f5b8a000640 1 -- 172.21.15.28:0/1512482283 wait complete. 2024-03-21T08:42:45.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:45.592 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-21T08:42:45.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:45.608 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-21T08:42:45.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-21T08:42:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 Processor -- start 2024-03-21T08:42:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 -- start start 2024-03-21T08:42:45.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 0x7f725406c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f725406cbe0 0x7f72540cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725406d050 con 0x7f725415e1e0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72540b3010 con 0x7f725406cbe0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7259a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725406d310 con 0x7f72540d0360 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7252a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f725415e1e0 0x7f72540cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41846/0 (socket says 172.21.15.28:41846) 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7252a00640 1 -- 172.21.15.28:0/1524927272 learned_addr learned my addr 172.21.15.28:0/1524927272 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f724be00640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 0x7f725406c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7253400640 1 --2- 172.21.15.28:0/1524927272 >> v2:172.21.15.151:3300/0 conn(0x7f725406cbe0 0x7f72540cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7252000640 1 -- 172.21.15.28:0/1524927272 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643988331 0 0) 0x7f725406d050 con 0x7f725415e1e0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7252000640 1 -- 172.21.15.28:0/1524927272 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238003530 con 0x7f725415e1e0 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7253400640 1 -- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 msgr2=0x7f725406c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7253400640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 0x7f725406c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7253400640 1 -- 172.21.15.28:0/1524927272 >> v1:172.21.15.28:6789/0 conn(0x7f725415e1e0 legacy=0x7f72540cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f7253400640 1 -- 172.21.15.28:0/1524927272 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725406d5f0 con 0x7f725406cbe0 2024-03-21T08:42:45.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.863+0000 7f724be00640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 0x7f725406c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7253400640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f72540cc4e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f724c009d90 tx=0x7f724c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2088929d4abee8b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7252000640 1 -- 172.21.15.28:0/1524927272 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c0b2070 con 0x7f725406cbe0 2024-03-21T08:42:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7252000640 1 -- 172.21.15.28:0/1524927272 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c0a5920 con 0x7f725406cbe0 2024-03-21T08:42:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7252000640 1 -- 172.21.15.28:0/1524927272 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c0a5cd0 con 0x7f725406cbe0 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 -- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 msgr2=0x7f72540cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f72540cc4e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f724c009d90 tx=0x7f724c01c970 comp rx=0 tx=0).stop 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 -- 172.21.15.28:0/1524927272 shutdown_connections 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f72540cc4e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 --2- 172.21.15.28:0/1524927272 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f72540d0360 0x7f725406c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 -- 172.21.15.28:0/1524927272 >> 172.21.15.28:0/1524927272 conn(0x7f72540d1ab0 msgr2=0x7f72540d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 -- 172.21.15.28:0/1524927272 shutdown_connections 2024-03-21T08:42:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.867+0000 7f7259a00640 1 -- 172.21.15.28:0/1524927272 wait complete. 2024-03-21T08:42:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 Processor -- start 2024-03-21T08:42:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- start start 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f7254154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f725415e1e0 0x7f7254148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540cd5c0 con 0x7f72540d0360 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540cd440 con 0x7f725406cbe0 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72540cd740 con 0x7f725415e1e0 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f7254154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38338/0 (socket says 172.21.15.28:38338) 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724be00640 1 -- 172.21.15.28:0/225654495 learned_addr learned my addr 172.21.15.28:0/225654495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7253400640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f725415e1e0 0x7f7254148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 -- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f725415e1e0 msgr2=0x7f7254148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f725415e1e0 0x7f7254148a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 -- 172.21.15.28:0/225654495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 msgr2=0x7f72541519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 -- 172.21.15.28:0/225654495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725414e200 con 0x7f725406cbe0 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724be00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7252a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f7254154140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f724400c400 tx=0x7f724400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244017070 con 0x7f725406cbe0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7244005cf0 con 0x7f725406cbe0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725414e4e0 con 0x7f725406cbe0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72440072f0 con 0x7f725406cbe0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.871+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725408bbf0 con 0x7f725406cbe0 2024-03-21T08:42:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725414e6b0 con 0x7f725406cbe0 2024-03-21T08:42:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f724401a070 con 0x7f725406cbe0 2024-03-21T08:42:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f724aa00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 0x7f7230069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7244085500 con 0x7f725406cbe0 2024-03-21T08:42:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f724be00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 0x7f7230069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.875+0000 7f724be00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 0x7f7230069fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72540ce4d0 tx=0x7f723c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:45.879+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f72440555b0 con 0x7f725406cbe0 2024-03-21T08:42:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.163+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7f725404f4f0 con 0x7f725406cbe0 2024-03-21T08:42:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.171+0000 7f724aa00640 1 -- 172.21.15.28:0/225654495 <== mon.1 v2:172.21.15.151:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v12) v1 ==== 210+0+167 (secure 0 0 0) 0x7f7244059650 con 0x7f725406cbe0 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow rw" 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-21T08:42:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 msgr2=0x7f7230069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 0x7f7230069fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f72540ce4d0 tx=0x7f723c009190 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 msgr2=0x7f7254154140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f7254154140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f724400c400 tx=0x7f724400c8c0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 shutdown_connections 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7230067b20 0x7f7230069fe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f725415e1e0 0x7f7254148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540d0360 0x7f72541519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 --2- 172.21.15.28:0/225654495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f725406cbe0 0x7f7254154140 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 >> 172.21.15.28:0/225654495 conn(0x7f72540d1ab0 msgr2=0x7f7254070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 shutdown_connections 2024-03-21T08:42:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.175+0000 7f7259a00640 1 -- 172.21.15.28:0/225654495 wait complete. 2024-03-21T08:42:46.204 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:46.204 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:46.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 Processor -- start 2024-03-21T08:42:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 -- start start 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0af060 0x7f8fcc0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fcc0b5520 0x7f8fcc15c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc15bf00 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0ce6e0 con 0x7f8fcc0b5520 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cdad0 con 0x7f8fcc0af060 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fcc0b5520 0x7f8fcc15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fcc0b5520 0x7f8fcc15c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38342/0 (socket says 172.21.15.28:38342) 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 -- 172.21.15.28:0/1801671 learned_addr learned my addr 172.21.15.28:0/1801671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fcbe00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0af060 0x7f8fcc0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 -- 172.21.15.28:0/1801671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0af060 msgr2=0x7f8fcc0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0af060 0x7f8fcc0cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 -- 172.21.15.28:0/1801671 >> v1:172.21.15.151:6789/0 conn(0x7f8fcc0b34f0 legacy=0x7f8fcc0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.395+0000 7f8fd0a00640 1 -- 172.21.15.28:0/1801671 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0cddb0 con 0x7f8fcc0b5520 2024-03-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd0a00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b5520 0x7f8fcc15c200 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009e50 tx=0x7f8fc401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d113c87d8f5c8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fcb400640 1 -- 172.21.15.28:0/1801671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40a5750 con 0x7f8fcc0b5520 2024-03-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fcb400640 1 -- 172.21.15.28:0/1801671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40a58f0 con 0x7f8fcc0b5520 2024-03-21T08:42:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fcb400640 1 -- 172.21.15.28:0/1801671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40a5cc0 con 0x7f8fcc0b5520 2024-03-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 -- 172.21.15.28:0/1801671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b5520 msgr2=0x7f8fcc15c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b5520 0x7f8fcc15c200 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009e50 tx=0x7f8fc401c9a0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 -- 172.21.15.28:0/1801671 shutdown_connections 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b5520 0x7f8fcc15c200 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/1801671 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0af060 0x7f8fcc0cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 -- 172.21.15.28:0/1801671 >> 172.21.15.28:0/1801671 conn(0x7f8fcc0cc230 msgr2=0x7f8fcc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 -- 172.21.15.28:0/1801671 shutdown_connections 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.399+0000 7f8fd2a00640 1 -- 172.21.15.28:0/1801671 wait complete. 2024-03-21T08:42:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 Processor -- start 2024-03-21T08:42:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- start start 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0b5520 0x7f8fcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fcc0ae260 0x7f8fcc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15d0d0 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15cf50 con 0x7f8fcc0ae260 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15d250 con 0x7f8fcc0b5520 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38356/0 (socket says 172.21.15.28:38356) 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 -- 172.21.15.28:0/802246766 learned_addr learned my addr 172.21.15.28:0/802246766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd0a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fcc0ae260 0x7f8fcc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fc3e00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0b5520 0x7f8fcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 -- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0b5520 msgr2=0x7f8fcc06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0b5520 0x7f8fcc06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 -- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fcc0ae260 msgr2=0x7f8fcc0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fcc0ae260 0x7f8fcc0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 -- 172.21.15.28:0/802246766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc06e330 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fcbe00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00d830 tx=0x7f8fbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc00b6c0 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc019070 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc06e610 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc00ba60 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.403+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc06ebd0 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc04f410 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8fbc01e460 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fc8c00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 0x7f8fa806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8fbc0867e0 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fc3e00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 0x7f8fa806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.407+0000 7f8fc3e00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 0x7f8fa806a010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc156c10 tx=0x7f8fb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.411+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8fbc056890 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.679+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fcc0ccd50 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fc8c00640 1 -- 172.21.15.28:0/802246766 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f8fbc084e20 con 0x7f8fcc0b34f0 2024-03-21T08:42:46.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:46.684 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 msgr2=0x7f8fa806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 0x7f8fa806a010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc156c10 tx=0x7f8fb4009040 comp rx=0 tx=0).stop 2024-03-21T08:42:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 msgr2=0x7f8fcc14a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00d830 tx=0x7f8fbc00dcf0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 shutdown_connections 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8fcc0ae260 0x7f8fcc0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8fa8067b50 0x7f8fa806a010 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8fcc0b5520 0x7f8fcc06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 --2- 172.21.15.28:0/802246766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fcc0b34f0 0x7f8fcc14a0b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 >> 172.21.15.28:0/802246766 conn(0x7f8fcc0cc230 msgr2=0x7f8fcc0b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 shutdown_connections 2024-03-21T08:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.683+0000 7f8fd2a00640 1 -- 172.21.15.28:0/802246766 wait complete. 2024-03-21T08:42:46.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:42:46.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 Processor -- start 2024-03-21T08:42:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 -- start start 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcbec15bd20 0x7fcbec0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec15fba0 con 0x7fcbec0b0e80 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec15fe80 con 0x7fcbec0cebd0 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbf3a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec160160 con 0x7fcbec15bd20 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbe3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcbec0b0e80 0x7fcbec0ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41848/0 (socket says 172.21.15.28:41848) 2024-03-21T08:42:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.851+0000 7fcbe3e00640 1 -- 172.21.15.28:0/2823353315 learned_addr learned my addr 172.21.15.28:0/2823353315 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 --2- 172.21.15.28:0/2823353315 >> v2:172.21.15.151:3300/0 conn(0x7fcbec15bd20 0x7fcbec0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906770037 0 0) 0x7fcbec15fba0 con 0x7fcbec0b0e80 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd0003530 con 0x7fcbec0b0e80 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065566976 0 0) 0x7fcbd0003530 con 0x7fcbec0b0e80 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 >> v2:172.21.15.151:3300/0 conn(0x7fcbec15bd20 msgr2=0x7fcbec0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 --2- 172.21.15.28:0/2823353315 >> v2:172.21.15.151:3300/0 conn(0x7fcbec15bd20 0x7fcbec0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 >> v1:172.21.15.151:6789/0 conn(0x7fcbec0cebd0 legacy=0x7fcbec08afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec160440 con 0x7fcbec0b0e80 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fcbd4001680 con 0x7fcbec0b0e80 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 >> v1:172.21.15.28:6789/0 conn(0x7fcbec0b0e80 legacy=0x7fcbec0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 0x7fcbd0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd0004810 0x7fcbd0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbd00051d0 0x7fcbd000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec160160 con 0x7fcbd00039c0 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd000c070 con 0x7fcbd0004810 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd000c270 con 0x7fcbd00051d0 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 0x7fcbd0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd0004810 0x7fcbd0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1a00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbd00051d0 0x7fcbd000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 -- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbd00051d0 msgr2=0x7fcbd000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbd00051d0 0x7fcbd000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 -- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd0004810 msgr2=0x7fcbd0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd0004810 0x7fcbd0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1000640 1 -- 172.21.15.28:0/2823353315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd0003530 con 0x7fcbd00039c0 2024-03-21T08:42:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.855+0000 7fcbf1a00640 1 -- 172.21.15.28:0/2823353315 reap_dead start 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf1000640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 0x7fcbd0003de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009ae0 tx=0x7fcbdc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8b3014efd86c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0096b0 con 0x7fcbd00039c0 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc0a6470 con 0x7fcbd00039c0 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbebe00640 1 -- 172.21.15.28:0/2823353315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0a6860 con 0x7fcbd00039c0 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 msgr2=0x7fcbd0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 0x7fcbd0003de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009ae0 tx=0x7fcbdc01ca50 comp rx=0 tx=0).stop 2024-03-21T08:42:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- 172.21.15.28:0/2823353315 shutdown_connections 2024-03-21T08:42:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/2823353315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbd00039c0 0x7fcbd0003de0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- 172.21.15.28:0/2823353315 >> 172.21.15.28:0/2823353315 conn(0x7fcbec0d1ab0 msgr2=0x7fcbec0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- 172.21.15.28:0/2823353315 shutdown_connections 2024-03-21T08:42:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- 172.21.15.28:0/2823353315 wait complete. 2024-03-21T08:42:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 Processor -- start 2024-03-21T08:42:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- start start 2024-03-21T08:42:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd00039c0 0x7fcbec159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 0x7fcbec15e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0b3050 con 0x7fcbec159d40 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec15c310 con 0x7fcbd00039c0 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.859+0000 7fcbf3a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0b31d0 con 0x7fcbec0c55e0 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38376/0 (socket says 172.21.15.28:38376) 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 -- 172.21.15.28:0/1240250757 learned_addr learned my addr 172.21.15.28:0/1240250757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf1000640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd00039c0 0x7fcbec159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf1a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 0x7fcbec15e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 -- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 msgr2=0x7fcbec15e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 0x7fcbec15e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 -- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd00039c0 msgr2=0x7fcbec159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd00039c0 0x7fcbec159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 -- 172.21.15.28:0/1240250757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0ce170 con 0x7fcbec159d40 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf1a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 0x7fcbec15e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40072c0 tx=0x7fcbd4007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400f8e0 con 0x7fcbec159d40 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4018070 con 0x7fcbec159d40 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec0af660 con 0x7fcbec159d40 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400fd00 con 0x7fcbec159d40 2024-03-21T08:42:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec0af900 con 0x7fcbec159d40 2024-03-21T08:42:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec15f2a0 con 0x7fcbec159d40 2024-03-21T08:42:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.863+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcbd401d460 con 0x7fcbec159d40 2024-03-21T08:42:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.867+0000 7fcbe9600640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 0x7fcbc806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.867+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcbd4086140 con 0x7fcbec159d40 2024-03-21T08:42:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.867+0000 7fcbf1000640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 0x7fcbc806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.867+0000 7fcbf1000640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 0x7fcbc806a010 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0ad000 tx=0x7fcbdc0b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:46.867+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcbd40567c0 con 0x7fcbec159d40 2024-03-21T08:42:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.031+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcbec04f4f0 con 0x7fcbec159d40 2024-03-21T08:42:47.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.031+0000 7fcbe9600640 1 -- 172.21.15.28:0/1240250757 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcbd40560c0 con 0x7fcbec159d40 2024-03-21T08:42:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 msgr2=0x7fcbc806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 0x7fcbc806a010 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0ad000 tx=0x7fcbdc0b2040 comp rx=0 tx=0).stop 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 msgr2=0x7fcbec0b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40072c0 tx=0x7fcbd4007780 comp rx=0 tx=0).stop 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 shutdown_connections 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcbc8067b50 0x7fcbc806a010 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcbec0c55e0 0x7fcbec15e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec159d40 0x7fcbec0b71b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 --2- 172.21.15.28:0/1240250757 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcbd00039c0 0x7fcbec159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 >> 172.21.15.28:0/1240250757 conn(0x7fcbec0d1ab0 msgr2=0x7fcbec15af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 shutdown_connections 2024-03-21T08:42:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.035+0000 7fcbf3a00640 1 -- 172.21.15.28:0/1240250757 wait complete. 2024-03-21T08:42:47.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.207+0000 7f1f20a00640 1 Processor -- start 2024-03-21T08:42:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.207+0000 7f1f20a00640 1 -- start start 2024-03-21T08:42:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f20a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f1c15aa30 0x7f1f1c15ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f20a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c15be50 con 0x7f1f1c0ce590 2024-03-21T08:42:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f20a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c15b890 con 0x7f1f1c15b350 2024-03-21T08:42:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f20a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cf930 con 0x7f1f1c15aa30 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f19e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f1c15aa30 0x7f1f1c15ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f11400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f1c0ce590 0x7f1f1c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41850/0 (socket says 172.21.15.28:41850) 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f11400640 1 -- 172.21.15.28:0/1899042188 learned_addr learned my addr 172.21.15.28:0/1899042188 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f19e00640 1 -- 172.21.15.28:0/1899042188 >> v1:172.21.15.151:6789/0 conn(0x7f1f1c15b350 legacy=0x7f1f1c0cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f19e00640 1 -- 172.21.15.28:0/1899042188 >> v1:172.21.15.28:6789/0 conn(0x7f1f1c0ce590 legacy=0x7f1f1c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f19e00640 1 -- 172.21.15.28:0/1899042188 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c15bb70 con 0x7f1f1c15aa30 2024-03-21T08:42:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.211+0000 7f1f19e00640 1 --2- 172.21.15.28:0/1899042188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c15ae10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0097f0 tx=0x7f1f0c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e69d8d79494871 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f19400640 1 -- 172.21.15.28:0/1899042188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0b1070 con 0x7f1f1c15aa30 2024-03-21T08:42:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f19400640 1 -- 172.21.15.28:0/1899042188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c0a8050 con 0x7f1f1c15aa30 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f19400640 1 -- 172.21.15.28:0/1899042188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0ac860 con 0x7f1f1c15aa30 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- 172.21.15.28:0/1899042188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 msgr2=0x7f1f1c15ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 --2- 172.21.15.28:0/1899042188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c15ae10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0097f0 tx=0x7f1f0c01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- 172.21.15.28:0/1899042188 shutdown_connections 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 --2- 172.21.15.28:0/1899042188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c15ae10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- 172.21.15.28:0/1899042188 >> 172.21.15.28:0/1899042188 conn(0x7f1f1c099a30 msgr2=0x7f1f1c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- 172.21.15.28:0/1899042188 shutdown_connections 2024-03-21T08:42:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- 172.21.15.28:0/1899042188 wait complete. 2024-03-21T08:42:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 Processor -- start 2024-03-21T08:42:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- start start 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 0x7f1f1c14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f1c15b350 0x7f1f1c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cc790 con 0x7f1f1c15aa30 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cc610 con 0x7f1f1c15b350 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f20a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0cc910 con 0x7f1f1c0ce590 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38390/0 (socket says 172.21.15.28:38390) 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f11400640 1 -- 172.21.15.28:0/3848438735 learned_addr learned my addr 172.21.15.28:0/3848438735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f19e00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 0x7f1f1c14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.215+0000 7f1f1a800640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f1c15b350 0x7f1f1c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 -- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 msgr2=0x7f1f1c14cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 0x7f1f1c14cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 -- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f1c15b350 msgr2=0x7f1f1c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f1c15b350 0x7f1f1c158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 -- 172.21.15.28:0/3848438735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c1572f0 con 0x7f1f1c15aa30 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f19e00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 0x7f1f1c14cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f11400640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c500 tx=0x7f1f0400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04017070 con 0x7f1f1c15aa30 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f04005cf0 con 0x7f1f1c15aa30 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f040072f0 con 0x7f1f1c15aa30 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c156d30 con 0x7f1f1c15aa30 2024-03-21T08:42:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c158460 con 0x7f1f1c15aa30 2024-03-21T08:42:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c09ea60 con 0x7f1f1c15aa30 2024-03-21T08:42:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1f0401a070 con 0x7f1f1c15aa30 2024-03-21T08:42:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 0x7f1ef8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.219+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1f04085450 con 0x7f1f1c15aa30 2024-03-21T08:42:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.223+0000 7f1f19e00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 0x7f1ef8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.223+0000 7f1f19e00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 0x7f1ef8069fe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0ae000 tx=0x7f1f0c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.223+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1f04055590 con 0x7f1f1c15aa30 2024-03-21T08:42:47.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.387+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f1c0c4220 con 0x7f1f1c15aa30 2024-03-21T08:42:47.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.387+0000 7f1f12a00640 1 -- 172.21.15.28:0/3848438735 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f04059630 con 0x7f1f1c15aa30 2024-03-21T08:42:47.390 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:47.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 msgr2=0x7f1ef8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 0x7f1ef8069fe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0ae000 tx=0x7f1f0c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 msgr2=0x7f1f1c159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c500 tx=0x7f1f0400c9c0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 shutdown_connections 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f1c15b350 0x7f1f1c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f1c15aa30 0x7f1f1c159840 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1ef8067b20 0x7f1ef8069fe0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 --2- 172.21.15.28:0/3848438735 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f1c0ce590 0x7f1f1c14cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 >> 172.21.15.28:0/3848438735 conn(0x7f1f1c099a30 msgr2=0x7f1f1c06ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 shutdown_connections 2024-03-21T08:42:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.391+0000 7f1f20a00640 1 -- 172.21.15.28:0/3848438735 wait complete. 2024-03-21T08:42:47.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.551+0000 7f4a1ca00640 1 Processor -- start 2024-03-21T08:42:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.551+0000 7f4a1ca00640 1 -- start start 2024-03-21T08:42:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a1ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a1815e1e0 0x7f4a180cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a1ca00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 0x7f4a180cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a1ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a180b3010 con 0x7f4a180d0190 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a1ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1806d310 con 0x7f4a1815e1e0 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a1ca00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1806d5f0 con 0x7f4a1806cdb0 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a0d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4a180d0190 0x7f4a1806c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41854/0 (socket says 172.21.15.28:41854) 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a0d400640 1 -- 172.21.15.28:0/3251040597 learned_addr learned my addr 172.21.15.28:0/3251040597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3251040597 >> v2:172.21.15.28:3300/0 conn(0x7f4a1815e1e0 0x7f4a180cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a16800640 1 --2- 172.21.15.28:0/3251040597 >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 0x7f4a180cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15e00640 1 -- 172.21.15.28:0/3251040597 >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 msgr2=0x7f4a180cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3251040597 >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 0x7f4a180cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15e00640 1 -- 172.21.15.28:0/3251040597 >> v1:172.21.15.28:6789/0 conn(0x7f4a180d0190 legacy=0x7f4a1806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15400640 1 -- 172.21.15.28:0/3251040597 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493609574 0 0) 0x7f4a180b3010 con 0x7f4a180d0190 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a15e00640 1 -- 172.21.15.28:0/3251040597 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1806d8d0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.555+0000 7f4a16800640 1 --2- 172.21.15.28:0/3251040597 >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 0x7f4a180cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3251040597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a180cfc50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4a180d0d20 tx=0x7f4a0801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80be640224383e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a15400640 1 -- 172.21.15.28:0/3251040597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080b0070 con 0x7f4a1815e1e0 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a15400640 1 -- 172.21.15.28:0/3251040597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a080ab600 con 0x7f4a1815e1e0 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a15400640 1 -- 172.21.15.28:0/3251040597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080ab9d0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3251040597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 msgr2=0x7f4a180cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3251040597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a180cfc50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4a180d0d20 tx=0x7f4a0801c3f0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3251040597 shutdown_connections 2024-03-21T08:42:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3251040597 >> v2:172.21.15.151:3300/0 conn(0x7f4a1806cdb0 0x7f4a180cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3251040597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a180cfc50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3251040597 >> 172.21.15.28:0/3251040597 conn(0x7f4a180d1ab0 msgr2=0x7f4a180d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3251040597 shutdown_connections 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3251040597 wait complete. 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 Processor -- start 2024-03-21T08:42:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.559+0000 7f4a1ca00640 1 -- start start 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1806cdb0 0x7f4a1815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 0x7f4a180ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a180cd5c0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a180cd440 con 0x7f4a1806cdb0 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a180cd740 con 0x7f4a180d0190 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38406/0 (socket says 172.21.15.28:38406) 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 -- 172.21.15.28:0/3121710294 learned_addr learned my addr 172.21.15.28:0/3121710294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:47.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1806cdb0 0x7f4a1815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0d400640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 0x7f4a180ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 -- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 msgr2=0x7f4a180ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 0x7f4a180ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 -- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1806cdb0 msgr2=0x7f4a1815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1806cdb0 0x7f4a1815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 -- 172.21.15.28:0/3121710294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1815b840 con 0x7f4a1815e1e0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0d400640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 0x7f4a180ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a16800640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009fd0 tx=0x7f4a1000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1001a070 con 0x7f4a1815e1e0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10003bc0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1815a540 con 0x7f4a1815e1e0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10004030 con 0x7f4a1815e1e0 2024-03-21T08:42:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1815ab00 con 0x7f4a1815e1e0 2024-03-21T08:42:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1804f4f0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.563+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4a1001d070 con 0x7f4a1815e1e0 2024-03-21T08:42:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.567+0000 7f4a0ea00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 0x7f49f4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.567+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4a100864d0 con 0x7f4a1815e1e0 2024-03-21T08:42:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.567+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 0x7f49f4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.567+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 0x7f49f4069fe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4a080ad000 tx=0x7f4a08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.575+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4a10010070 con 0x7f4a1815e1e0 2024-03-21T08:42:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.739+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a180d2830 con 0x7f4a1815e1e0 2024-03-21T08:42:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.739+0000 7f4a0ea00640 1 -- 172.21.15.28:0/3121710294 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4a10056500 con 0x7f4a1815e1e0 2024-03-21T08:42:47.741 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.739+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 msgr2=0x7f49f4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.739+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 0x7f49f4069fe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4a080ad000 tx=0x7f4a08008040 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 msgr2=0x7f4a1814fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009fd0 tx=0x7f4a1000d560 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 shutdown_connections 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a1815e1e0 0x7f4a1814fa50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49f4067b20 0x7f49f4069fe0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4a180d0190 0x7f4a180ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 --2- 172.21.15.28:0/3121710294 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a1806cdb0 0x7f4a1815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 >> 172.21.15.28:0/3121710294 conn(0x7f4a180d1ab0 msgr2=0x7f4a180c9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 shutdown_connections 2024-03-21T08:42:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.743+0000 7f4a1ca00640 1 -- 172.21.15.28:0/3121710294 wait complete. 2024-03-21T08:42:47.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:42:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 Processor -- start 2024-03-21T08:42:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 -- start start 2024-03-21T08:42:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ced70 0x7fb0c815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0c80b31f0 0x7fb0c80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80d0c50 0x7fb0c80d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c80d1600 con 0x7fb0c80b31f0 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c8161a20 con 0x7fb0c80ced70 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0cd600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c815e9e0 con 0x7fb0c80d0c50 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0c80b31f0 0x7fb0c80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0c80b31f0 0x7fb0c80d1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38410/0 (socket says 172.21.15.28:38410) 2024-03-21T08:42:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 -- 172.21.15.28:0/1442351078 learned_addr learned my addr 172.21.15.28:0/1442351078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0c7400640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80d0c50 0x7fb0c80d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1442351078 >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ced70 0x7fb0c815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 -- 172.21.15.28:0/1442351078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80d0c50 msgr2=0x7fb0c80d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80d0c50 0x7fb0c80d10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 -- 172.21.15.28:0/1442351078 >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ced70 msgr2=0x7fb0c815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 --2- 172.21.15.28:0/1442351078 >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ced70 0x7fb0c815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.911+0000 7fb0be000640 1 -- 172.21.15.28:0/1442351078 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c815ecc0 con 0x7fb0c80b31f0 2024-03-21T08:42:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0be000640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b31f0 0x7fb0c80d1a60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80d4f40 tx=0x7fb0b001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f058d47ef1c77a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0c6000640 1 -- 172.21.15.28:0/1442351078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00aa5f0 con 0x7fb0c80b31f0 2024-03-21T08:42:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0c6000640 1 -- 172.21.15.28:0/1442351078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b00af070 con 0x7fb0c80b31f0 2024-03-21T08:42:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0c6000640 1 -- 172.21.15.28:0/1442351078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00aa9b0 con 0x7fb0c80b31f0 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 -- 172.21.15.28:0/1442351078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b31f0 msgr2=0x7fb0c80d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b31f0 0x7fb0c80d1a60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80d4f40 tx=0x7fb0b001c4f0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 -- 172.21.15.28:0/1442351078 shutdown_connections 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80d0c50 0x7fb0c80d10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 --2- 172.21.15.28:0/1442351078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80b31f0 0x7fb0c80d1a60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 --2- 172.21.15.28:0/1442351078 >> v2:172.21.15.151:3300/0 conn(0x7fb0c80ced70 0x7fb0c815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 -- 172.21.15.28:0/1442351078 >> 172.21.15.28:0/1442351078 conn(0x7fb0c8072aa0 msgr2=0x7fb0c8072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 -- 172.21.15.28:0/1442351078 shutdown_connections 2024-03-21T08:42:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.915+0000 7fb0cd600640 1 -- 172.21.15.28:0/1442351078 wait complete. 2024-03-21T08:42:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 Processor -- start 2024-03-21T08:42:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- start start 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b31f0 0x7fb0c80c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ced70 0x7fb0c8153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80d2120 con 0x7fb0c80d0c50 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80d1fa0 con 0x7fb0c80ced70 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80cf150 con 0x7fb0c80b31f0 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38418/0 (socket says 172.21.15.28:38418) 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 -- 172.21.15.28:0/3600178329 learned_addr learned my addr 172.21.15.28:0/3600178329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b31f0 0x7fb0c80c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0be000640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ced70 0x7fb0c8153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 -- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b31f0 msgr2=0x7fb0c80c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b31f0 0x7fb0c80c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 -- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ced70 msgr2=0x7fb0c8153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ced70 0x7fb0c8153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 -- 172.21.15.28:0/3600178329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c8150550 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000d800 tx=0x7fb0c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000b6c0 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0019070 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000baa0 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c8150830 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8150df0 con 0x7fb0c80d0c50 2024-03-21T08:42:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.919+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c808afc0 con 0x7fb0c80d0c50 2024-03-21T08:42:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.923+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb0c000bc40 con 0x7fb0c80d0c50 2024-03-21T08:42:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.923+0000 7fb0bf400640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 0x7fb0a4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.923+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb0c0086580 con 0x7fb0c80d0c50 2024-03-21T08:42:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.923+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 0x7fb0a4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.923+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 0x7fb0a4069fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8148a60 tx=0x7fb0b8009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:47.931+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb0c0056630 con 0x7fb0c80d0c50 2024-03-21T08:42:48.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fb0c804f4f0 con 0x7fb0c80d0c50 2024-03-21T08:42:48.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0bf400640 1 -- 172.21.15.28:0/3600178329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7fb0c005a6d0 con 0x7fb0c80d0c50 2024-03-21T08:42:48.096 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 msgr2=0x7fb0a4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 0x7fb0a4069fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8148a60 tx=0x7fb0b8009830 comp rx=0 tx=0).stop 2024-03-21T08:42:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 msgr2=0x7fb0c81541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000d800 tx=0x7fb0c000dcc0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 shutdown_connections 2024-03-21T08:42:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c80d0c50 0x7fb0c81541f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb0c80ced70 0x7fb0c8153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb0a4067b20 0x7fb0a4069fe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 --2- 172.21.15.28:0/3600178329 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb0c80b31f0 0x7fb0c80c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 >> 172.21.15.28:0/3600178329 conn(0x7fb0c8072aa0 msgr2=0x7fb0c8160c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 shutdown_connections 2024-03-21T08:42:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.095+0000 7fb0cd600640 1 -- 172.21.15.28:0/3600178329 wait complete. 2024-03-21T08:42:48.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 Processor -- start 2024-03-21T08:42:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 -- start start 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad8c06d3a0 0x7fad8c0d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 0x7fad8c0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c0b48a0 con 0x7fad8c15df10 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c0cc0a0 con 0x7fad8c06d3a0 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad90c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c0cc940 con 0x7fad8c0d0e00 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad89e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad8c15df10 0x7fad8c06cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41860/0 (socket says 172.21.15.28:41860) 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad89400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad8c06d3a0 0x7fad8c0d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad89e00640 1 -- 172.21.15.28:0/33964804 learned_addr learned my addr 172.21.15.28:0/33964804 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.283+0000 7fad8a800640 1 --2- 172.21.15.28:0/33964804 >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 0x7fad8c0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad89400640 1 -- 172.21.15.28:0/33964804 >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 msgr2=0x7fad8c0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad89400640 1 --2- 172.21.15.28:0/33964804 >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 0x7fad8c0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad89400640 1 -- 172.21.15.28:0/33964804 >> v1:172.21.15.28:6789/0 conn(0x7fad8c15df10 legacy=0x7fad8c06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad88a00640 1 -- 172.21.15.28:0/33964804 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103191393 0 0) 0x7fad7c001120 con 0x7fad8c15df10 2024-03-21T08:42:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad89400640 1 -- 172.21.15.28:0/33964804 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0d1620 con 0x7fad8c06d3a0 2024-03-21T08:42:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad8a800640 1 --2- 172.21.15.28:0/33964804 >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 0x7fad8c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad89400640 1 --2- 172.21.15.28:0/33964804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c06d3a0 0x7fad8c0d08c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0b7ab0 tx=0x7fad7801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e4349910e238c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad88a00640 1 -- 172.21.15.28:0/33964804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad780b0070 con 0x7fad8c06d3a0 2024-03-21T08:42:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad88a00640 1 -- 172.21.15.28:0/33964804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad780ab600 con 0x7fad8c06d3a0 2024-03-21T08:42:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad88a00640 1 -- 172.21.15.28:0/33964804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad780ab9d0 con 0x7fad8c06d3a0 2024-03-21T08:42:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad90c00640 1 -- 172.21.15.28:0/33964804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c06d3a0 msgr2=0x7fad8c0d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.287+0000 7fad90c00640 1 --2- 172.21.15.28:0/33964804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c06d3a0 0x7fad8c0d08c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0b7ab0 tx=0x7fad7801c3f0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- 172.21.15.28:0/33964804 shutdown_connections 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 --2- 172.21.15.28:0/33964804 >> v2:172.21.15.151:3300/0 conn(0x7fad8c0d0e00 0x7fad8c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 --2- 172.21.15.28:0/33964804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c06d3a0 0x7fad8c0d08c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- 172.21.15.28:0/33964804 >> 172.21.15.28:0/33964804 conn(0x7fad8c0b3540 msgr2=0x7fad8c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- 172.21.15.28:0/33964804 shutdown_connections 2024-03-21T08:42:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- 172.21.15.28:0/33964804 wait complete. 2024-03-21T08:42:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 Processor -- start 2024-03-21T08:42:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- start start 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 0x7fad8c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 0x7fad8c0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c0cdca0 con 0x7fad8c0d0e00 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c0cdb20 con 0x7fad8c15df10 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c0cde20 con 0x7fad8c06d3a0 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38434/0 (socket says 172.21.15.28:38434) 2024-03-21T08:42:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 -- 172.21.15.28:0/1866728636 learned_addr learned my addr 172.21.15.28:0/1866728636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad8a800640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 0x7fad8c0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89e00640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 0x7fad8c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 -- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 msgr2=0x7fad8c15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 0x7fad8c15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 -- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 msgr2=0x7fad8c0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 0x7fad8c0c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 -- 172.21.15.28:0/1866728636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0c8d30 con 0x7fad8c0d0e00 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad8a800640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 0x7fad8c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89e00640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 0x7fad8c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad89400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fad780ae040 tx=0x7fad78019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad780b8020 con 0x7fad8c0d0e00 2024-03-21T08:42:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad780b4410 con 0x7fad8c0d0e00 2024-03-21T08:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.291+0000 7fad90c00640 1 -- 172.21.15.28:0/1866728636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c0c9010 con 0x7fad8c0d0e00 2024-03-21T08:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad780b0040 con 0x7fad8c0d0e00 2024-03-21T08:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad90c00640 1 -- 172.21.15.28:0/1866728636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c0c95d0 con 0x7fad8c0d0e00 2024-03-21T08:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c04f4f0 con 0x7fad8c0d0e00 2024-03-21T08:42:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fad780bb020 con 0x7fad8c0d0e00 2024-03-21T08:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad82000640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 0x7fad6c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fad7811fdf0 con 0x7fad8c0d0e00 2024-03-21T08:42:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.295+0000 7fad89e00640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 0x7fad6c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.299+0000 7fad89e00640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 0x7fad6c069fe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad7c007fd0 tx=0x7fad7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.299+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fad780c2030 con 0x7fad8c0d0e00 2024-03-21T08:42:48.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.475+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fad8c06cea0 con 0x7fad8c0d0e00 2024-03-21T08:42:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.475+0000 7fad82000640 1 -- 172.21.15.28:0/1866728636 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7fad78016070 con 0x7fad8c0d0e00 2024-03-21T08:42:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 msgr2=0x7fad6c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 0x7fad6c069fe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad7c007fd0 tx=0x7fad7c009040 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 msgr2=0x7fad8c1f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fad780ae040 tx=0x7fad78019190 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 shutdown_connections 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fad8c15df10 0x7fad8c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad8c0d0e00 0x7fad8c1f68d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fad6c067b20 0x7fad6c069fe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 --2- 172.21.15.28:0/1866728636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fad8c06d3a0 0x7fad8c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 >> 172.21.15.28:0/1866728636 conn(0x7fad8c0b3540 msgr2=0x7fad8c070860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 shutdown_connections 2024-03-21T08:42:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.479+0000 7fad77400640 1 -- 172.21.15.28:0/1866728636 wait complete. 2024-03-21T08:42:48.502 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:48.502 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:48.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 Processor -- start 2024-03-21T08:42:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 -- start start 2024-03-21T08:42:48.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3d015f260 0x7fb3d00cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 0x7fb3d0099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00cdb50 con 0x7fb3d009a0c0 2024-03-21T08:42:48.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d00c5530 con 0x7fb3d015f260 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d7e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d00cde30 con 0x7fb3d00cd5d0 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3d015f260 0x7fb3d00cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3d015f260 0x7fb3d00cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38440/0 (socket says 172.21.15.28:38440) 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 -- 172.21.15.28:0/3931719740 learned_addr learned my addr 172.21.15.28:0/3931719740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3931719740 >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 0x7fb3d0099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 -- 172.21.15.28:0/3931719740 >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 msgr2=0x7fb3d0099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 --2- 172.21.15.28:0/3931719740 >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 0x7fb3d0099b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 -- 172.21.15.28:0/3931719740 >> v1:172.21.15.151:6789/0 conn(0x7fb3d009a0c0 legacy=0x7fb3d00cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d5400640 1 -- 172.21.15.28:0/3931719740 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00c5780 con 0x7fb3d015f260 2024-03-21T08:42:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.695+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3931719740 >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 0x7fb3d0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d5400640 1 --2- 172.21.15.28:0/3931719740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d015f260 0x7fb3d00cd090 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00730b0 tx=0x7fb3c401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc04500c8e38159 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3cfe00640 1 -- 172.21.15.28:0/3931719740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40b2070 con 0x7fb3d015f260 2024-03-21T08:42:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3cfe00640 1 -- 172.21.15.28:0/3931719740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c40a9070 con 0x7fb3d015f260 2024-03-21T08:42:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3cfe00640 1 -- 172.21.15.28:0/3931719740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40ad880 con 0x7fb3d015f260 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3931719740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d015f260 msgr2=0x7fb3d00cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3931719740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d015f260 0x7fb3d00cd090 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00730b0 tx=0x7fb3c401c450 comp rx=0 tx=0).stop 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3931719740 shutdown_connections 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3931719740 >> v2:172.21.15.151:3300/0 conn(0x7fb3d00cd5d0 0x7fb3d0099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3931719740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d015f260 0x7fb3d00cd090 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3931719740 >> 172.21.15.28:0/3931719740 conn(0x7fb3d00b34d0 msgr2=0x7fb3d00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3931719740 shutdown_connections 2024-03-21T08:42:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.699+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3931719740 wait complete. 2024-03-21T08:42:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 Processor -- start 2024-03-21T08:42:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- start start 2024-03-21T08:42:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 0x7fb3d0156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3d015f260 0x7fb3d0156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0168bc0 con 0x7fb3d009a0c0 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0168a40 con 0x7fb3d00cd5d0 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0168d40 con 0x7fb3d015f260 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38442/0 (socket says 172.21.15.28:38442) 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 -- 172.21.15.28:0/3985747698 learned_addr learned my addr 172.21.15.28:0/3985747698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 0x7fb3d0156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3d015f260 0x7fb3d0156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 -- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3d015f260 msgr2=0x7fb3d0156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3d015f260 0x7fb3d0156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 -- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 msgr2=0x7fb3d0156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 0x7fb3d0156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 -- 172.21.15.28:0/3985747698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0159b70 con 0x7fb3d009a0c0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 0x7fb3d0156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d5400640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb3c401a080 tx=0x7fb3c40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40b2040 con 0x7fb3d009a0c0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c40bb070 con 0x7fb3d009a0c0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d0159e50 con 0x7fb3d009a0c0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40b6680 con 0x7fb3d009a0c0 2024-03-21T08:42:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.703+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d015a410 con 0x7fb3d009a0c0 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d004f410 con 0x7fb3d009a0c0 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb3c40ab070 con 0x7fb3d009a0c0 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3cd600640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 0x7fb3b0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb3c411fd20 con 0x7fb3d009a0c0 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 0x7fb3b0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.707+0000 7fb3d4a00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 0x7fb3b0069fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc006fd0 tx=0x7fb3bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.711+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3c40160d0 con 0x7fb3d009a0c0 2024-03-21T08:42:48.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.911+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3d009b060 con 0x7fb3d009a0c0 2024-03-21T08:42:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.911+0000 7fb3cd600640 1 -- 172.21.15.28:0/3985747698 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7fb3c40f0430 con 0x7fb3d009a0c0 2024-03-21T08:42:48.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 msgr2=0x7fb3b0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 0x7fb3b0069fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc006fd0 tx=0x7fb3bc008040 comp rx=0 tx=0).stop 2024-03-21T08:42:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 msgr2=0x7fb3d0165040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb3c401a080 tx=0x7fb3c40191f0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 shutdown_connections 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb3b0067b20 0x7fb3b0069fe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb3d015f260 0x7fb3d0156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb3d00cd5d0 0x7fb3d0156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/3985747698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3d009a0c0 0x7fb3d0165040 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 >> 172.21.15.28:0/3985747698 conn(0x7fb3d00b34d0 msgr2=0x7fb3d00713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 shutdown_connections 2024-03-21T08:42:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:48.915+0000 7fb3d7e00640 1 -- 172.21.15.28:0/3985747698 wait complete. 2024-03-21T08:42:48.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:42:48.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.079+0000 7fcde0000640 1 Processor -- start 2024-03-21T08:42:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.079+0000 7fcde0000640 1 -- start start 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcde0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcde0000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 0x7fcdd80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcde0000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd815fba0 con 0x7fcdd80cea00 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcde0000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd808bfd0 con 0x7fcdd80b0e80 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcde0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80cee50 con 0x7fcdd815bd20 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcdde000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcdd80cea00 0x7fcdd80b19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41864/0 (socket says 172.21.15.28:41864) 2024-03-21T08:42:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcdde000640 1 -- 172.21.15.28:0/1597248636 learned_addr learned my addr 172.21.15.28:0/1597248636 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddcc00640 1 --2- 172.21.15.28:0/1597248636 >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 0x7fcdd80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddd600640 1 --2- 172.21.15.28:0/1597248636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261659778 0 0) 0x7fcdd815fba0 con 0x7fcdd80cea00 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc0003530 con 0x7fcdd80cea00 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddd600640 1 -- 172.21.15.28:0/1597248636 >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 msgr2=0x7fcdd80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddd600640 1 --2- 172.21.15.28:0/1597248636 >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 0x7fcdd80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddd600640 1 -- 172.21.15.28:0/1597248636 >> v1:172.21.15.28:6789/0 conn(0x7fcdd80cea00 legacy=0x7fcdd80b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494061593 0 0) 0x7fcdc0003530 con 0x7fcdd80cea00 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddd600640 1 -- 172.21.15.28:0/1597248636 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd815fe80 con 0x7fcdd815bd20 2024-03-21T08:42:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.083+0000 7fcddcc00640 1 --2- 172.21.15.28:0/1597248636 >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 0x7fcdd80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcddd600640 1 --2- 172.21.15.28:0/1597248636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b2c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0097c0 tx=0x7fcdcc01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75fb9b665b422065 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0b0070 con 0x7fcdd815bd20 2024-03-21T08:42:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc0ab400 con 0x7fcdd815bd20 2024-03-21T08:42:49.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcdd7e00640 1 -- 172.21.15.28:0/1597248636 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0001a00 con 0x7fcdd815bd20 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 -- 172.21.15.28:0/1597248636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 msgr2=0x7fcdd80b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 --2- 172.21.15.28:0/1597248636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b2c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0097c0 tx=0x7fcdcc01c3c0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 -- 172.21.15.28:0/1597248636 shutdown_connections 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 --2- 172.21.15.28:0/1597248636 >> v2:172.21.15.151:3300/0 conn(0x7fcdd80b0e80 0x7fcdd80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 --2- 172.21.15.28:0/1597248636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b2c40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 -- 172.21.15.28:0/1597248636 >> 172.21.15.28:0/1597248636 conn(0x7fcdd80d1ab0 msgr2=0x7fcdd80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 -- 172.21.15.28:0/1597248636 shutdown_connections 2024-03-21T08:42:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.087+0000 7fcde0000640 1 -- 172.21.15.28:0/1597248636 wait complete. 2024-03-21T08:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 Processor -- start 2024-03-21T08:42:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- start start 2024-03-21T08:42:49.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdd80b0e80 0x7fcdd80c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80cc6a0 con 0x7fcdd80cea00 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80cc520 con 0x7fcdd80b0e80 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd815c100 con 0x7fcdd815bd20 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38444/0 (socket says 172.21.15.28:38444) 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 -- 172.21.15.28:0/4120374313 learned_addr learned my addr 172.21.15.28:0/4120374313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcdde000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddd600640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdd80b0e80 0x7fcdd80c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 -- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 msgr2=0x7fcdd80b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 -- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdd80b0e80 msgr2=0x7fcdd80c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdd80b0e80 0x7fcdd80c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 -- 172.21.15.28:0/4120374313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80c6f00 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcdde000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcddcc00640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e980 tx=0x7fcdc400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4017070 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400ca00 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400cdd0 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80c71e0 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd80c77a0 con 0x7fcdd80cea00 2024-03-21T08:42:49.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.091+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd804f4f0 con 0x7fcdd80cea00 2024-03-21T08:42:49.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.095+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcdc401a070 con 0x7fcdd80cea00 2024-03-21T08:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.095+0000 7fcdd5600640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 0x7fcdb806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.095+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdc4085fe0 con 0x7fcdd80cea00 2024-03-21T08:42:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.095+0000 7fcddd600640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 0x7fcdb806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.095+0000 7fcddd600640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 0x7fcdb806a030 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0ad000 tx=0x7fcdcc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.103+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcdc4056140 con 0x7fcdd80cea00 2024-03-21T08:42:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.267+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcdd8158a60 con 0x7fcdd80cea00 2024-03-21T08:42:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.267+0000 7fcdd5600640 1 -- 172.21.15.28:0/4120374313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcdc405a1e0 con 0x7fcdd80cea00 2024-03-21T08:42:49.271 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 msgr2=0x7fcdb806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 0x7fcdb806a030 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0ad000 tx=0x7fcdcc0a6000 comp rx=0 tx=0).stop 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 msgr2=0x7fcdd8152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e980 tx=0x7fcdc400ee40 comp rx=0 tx=0).stop 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 shutdown_connections 2024-03-21T08:42:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdb8067b70 0x7fcdb806a030 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdd815bd20 0x7fcdd80b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd80cea00 0x7fcdd8152a70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 --2- 172.21.15.28:0/4120374313 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdd80b0e80 0x7fcdd80c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 >> 172.21.15.28:0/4120374313 conn(0x7fcdd80d1ab0 msgr2=0x7fcdd80cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 shutdown_connections 2024-03-21T08:42:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.271+0000 7fcde0000640 1 -- 172.21.15.28:0/4120374313 wait complete. 2024-03-21T08:42:49.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 Processor -- start 2024-03-21T08:42:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 -- start start 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd2600b2970 0x7fd2600ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2600cd250 0x7fd2600cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2600cd9b0 con 0x7fd26015bff0 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd260099e30 con 0x7fd2600cd250 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd266e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26009a110 con 0x7fd2600b2970 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2600cd250 0x7fd2600cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2600cd250 0x7fd2600cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38456/0 (socket says 172.21.15.28:38456) 2024-03-21T08:42:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 -- 172.21.15.28:0/576494953 learned_addr learned my addr 172.21.15.28:0/576494953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd257e00640 1 --2- 172.21.15.28:0/576494953 >> v2:172.21.15.151:3300/0 conn(0x7fd2600b2970 0x7fd2600ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 -- 172.21.15.28:0/576494953 >> v2:172.21.15.151:3300/0 conn(0x7fd2600b2970 msgr2=0x7fd2600ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 --2- 172.21.15.28:0/576494953 >> v2:172.21.15.151:3300/0 conn(0x7fd2600b2970 0x7fd2600ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 -- 172.21.15.28:0/576494953 >> v1:172.21.15.151:6789/0 conn(0x7fd26015bff0 legacy=0x7fd2600cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.435+0000 7fd264e00640 1 -- 172.21.15.28:0/576494953 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26009a3f0 con 0x7fd2600cd250 2024-03-21T08:42:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd264e00640 1 --2- 172.21.15.28:0/576494953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2600cd250 0x7fd2600cf700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd2600b35f0 tx=0x7fd25801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37eb65a19f378e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd25f400640 1 -- 172.21.15.28:0/576494953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580a57d0 con 0x7fd2600cd250 2024-03-21T08:42:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd25f400640 1 -- 172.21.15.28:0/576494953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580a5970 con 0x7fd2600cd250 2024-03-21T08:42:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd25f400640 1 -- 172.21.15.28:0/576494953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580a5d40 con 0x7fd2600cd250 2024-03-21T08:42:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 -- 172.21.15.28:0/576494953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2600cd250 msgr2=0x7fd2600cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 --2- 172.21.15.28:0/576494953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2600cd250 0x7fd2600cf700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd2600b35f0 tx=0x7fd25801c9f0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 -- 172.21.15.28:0/576494953 shutdown_connections 2024-03-21T08:42:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 --2- 172.21.15.28:0/576494953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2600cd250 0x7fd2600cf700 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 --2- 172.21.15.28:0/576494953 >> v2:172.21.15.151:3300/0 conn(0x7fd2600b2970 0x7fd2600ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 -- 172.21.15.28:0/576494953 >> 172.21.15.28:0/576494953 conn(0x7fd2600b3620 msgr2=0x7fd2600b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 -- 172.21.15.28:0/576494953 shutdown_connections 2024-03-21T08:42:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.439+0000 7fd266e00640 1 -- 172.21.15.28:0/576494953 wait complete. 2024-03-21T08:42:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 Processor -- start 2024-03-21T08:42:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- start start 2024-03-21T08:42:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 0x7fd2600cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2600cd250 0x7fd2600afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600d07e0 con 0x7fd26015bff0 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600d0660 con 0x7fd2600cd250 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600d0960 con 0x7fd2600b2970 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38462/0 (socket says 172.21.15.28:38462) 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 -- 172.21.15.28:0/1991466669 learned_addr learned my addr 172.21.15.28:0/1991466669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd257e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2600cd250 0x7fd2600afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd25fe00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 0x7fd2600cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 -- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 msgr2=0x7fd2600cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 0x7fd2600cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 -- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2600cd250 msgr2=0x7fd2600afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2600cd250 0x7fd2600afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 -- 172.21.15.28:0/1991466669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600aea80 con 0x7fd26015bff0 2024-03-21T08:42:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd25fe00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 0x7fd2600cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd264e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd2580af040 tx=0x7fd258019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580bc020 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580b88a0 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2600aed60 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580b8c90 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.443+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2600af320 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26009faf0 con 0x7fd26015bff0 2024-03-21T08:42:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd2580bf020 con 0x7fd26015bff0 2024-03-21T08:42:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd25cc00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 0x7fd23c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd258121420 con 0x7fd26015bff0 2024-03-21T08:42:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd25fe00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 0x7fd23c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.447+0000 7fd25fe00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 0x7fd23c06a010 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd2601f9dd0 tx=0x7fd250009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.451+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd2580c6030 con 0x7fd26015bff0 2024-03-21T08:42:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.619+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd26004f4f0 con 0x7fd26015bff0 2024-03-21T08:42:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.619+0000 7fd25cc00640 1 -- 172.21.15.28:0/1991466669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd2580165e0 con 0x7fd26015bff0 2024-03-21T08:42:49.623 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 msgr2=0x7fd23c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 0x7fd23c06a010 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd2601f9dd0 tx=0x7fd250009040 comp rx=0 tx=0).stop 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 msgr2=0x7fd2600b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd2580af040 tx=0x7fd258019790 comp rx=0 tx=0).stop 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 shutdown_connections 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd26015bff0 0x7fd2600b06d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2600cd250 0x7fd2600afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd23c067b50 0x7fd23c06a010 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 --2- 172.21.15.28:0/1991466669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2600b2970 0x7fd2600cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 >> 172.21.15.28:0/1991466669 conn(0x7fd2600b3620 msgr2=0x7fd260070910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 shutdown_connections 2024-03-21T08:42:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.623+0000 7fd266e00640 1 -- 172.21.15.28:0/1991466669 wait complete. 2024-03-21T08:42:49.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.783+0000 7fb754800640 1 Processor -- start 2024-03-21T08:42:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 -- start start 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb74c099c90 0x7fb74c09a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c09b1e0 con 0x7fb74c15e2a0 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c09a630 con 0x7fb74c099c90 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb754800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0d0930 con 0x7fb74c0cc540 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb74c099c90 0x7fb74c09a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb74c099c90 0x7fb74c09a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38464/0 (socket says 172.21.15.28:38464) 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 -- 172.21.15.28:0/664218760 learned_addr learned my addr 172.21.15.28:0/664218760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb752800640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 -- 172.21.15.28:0/664218760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 msgr2=0x7fb74c0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 -- 172.21.15.28:0/664218760 >> v1:172.21.15.151:6789/0 conn(0x7fb74c15e2a0 legacy=0x7fb74c09aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb751400640 1 -- 172.21.15.28:0/664218760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0d0c10 con 0x7fb74c099c90 2024-03-21T08:42:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.787+0000 7fb752800640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c0cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb751400640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c09a0f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009870 tx=0x7fb73c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab047444b6cb245 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb750a00640 1 -- 172.21.15.28:0/664218760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0b1070 con 0x7fb74c099c90 2024-03-21T08:42:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb750a00640 1 -- 172.21.15.28:0/664218760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c0ac620 con 0x7fb74c099c90 2024-03-21T08:42:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb750a00640 1 -- 172.21.15.28:0/664218760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0ac9f0 con 0x7fb74c099c90 2024-03-21T08:42:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- 172.21.15.28:0/664218760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 msgr2=0x7fb74c09a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c09a0f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009870 tx=0x7fb73c01c4d0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- 172.21.15.28:0/664218760 shutdown_connections 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c0cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 --2- 172.21.15.28:0/664218760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c09a0f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- 172.21.15.28:0/664218760 >> 172.21.15.28:0/664218760 conn(0x7fb74c0b3620 msgr2=0x7fb74c0b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- 172.21.15.28:0/664218760 shutdown_connections 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- 172.21.15.28:0/664218760 wait complete. 2024-03-21T08:42:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 Processor -- start 2024-03-21T08:42:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.791+0000 7fb754800640 1 -- start start 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c14dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb74c15e2a0 0x7fb74c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c15e680 con 0x7fb74c099c90 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c16aee0 con 0x7fb74c15e2a0 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0b5c10 con 0x7fb74c0cc540 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38468/0 (socket says 172.21.15.28:38468) 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 -- 172.21.15.28:0/863675254 learned_addr learned my addr 172.21.15.28:0/863675254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb752800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb74c15e2a0 0x7fb74c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751400640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c14dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 -- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 msgr2=0x7fb74c14dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c14dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 -- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb74c15e2a0 msgr2=0x7fb74c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb74c15e2a0 0x7fb74c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 -- 172.21.15.28:0/863675254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0b5ef0 con 0x7fb74c099c90 2024-03-21T08:42:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb751e00640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c420 tx=0x7fb74000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400052d0 con 0x7fb74c099c90 2024-03-21T08:42:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740005470 con 0x7fb74c099c90 2024-03-21T08:42:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c0b61d0 con 0x7fb74c099c90 2024-03-21T08:42:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740007260 con 0x7fb74c099c90 2024-03-21T08:42:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c14e290 con 0x7fb74c099c90 2024-03-21T08:42:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.795+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c09faf0 con 0x7fb74c099c90 2024-03-21T08:42:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.799+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb740007400 con 0x7fb74c099c90 2024-03-21T08:42:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.799+0000 7fb746000640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 0x7fb730069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.799+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb7400856f0 con 0x7fb74c099c90 2024-03-21T08:42:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.799+0000 7fb751400640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 0x7fb730069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.799+0000 7fb751400640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 0x7fb730069fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0ae000 tx=0x7fb73c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.803+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb74c09b1e0 con 0x7fb74c099c90 2024-03-21T08:42:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.963+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb74c04f4f0 con 0x7fb74c099c90 2024-03-21T08:42:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.963+0000 7fb746000640 1 -- 172.21.15.28:0/863675254 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb740055720 con 0x7fb74c099c90 2024-03-21T08:42:49.967 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 msgr2=0x7fb730069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 0x7fb730069fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0ae000 tx=0x7fb73c008040 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 msgr2=0x7fb74c0b69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb74000c420 tx=0x7fb74000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 shutdown_connections 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb74c15e2a0 0x7fb74c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb730067b20 0x7fb730069fe0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb74c0cc540 0x7fb74c14dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 --2- 172.21.15.28:0/863675254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c099c90 0x7fb74c0b69d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 >> 172.21.15.28:0/863675254 conn(0x7fb74c0b3620 msgr2=0x7fb74c06fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 shutdown_connections 2024-03-21T08:42:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:49.967+0000 7fb754800640 1 -- 172.21.15.28:0/863675254 wait complete. 2024-03-21T08:42:49.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:42:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 Processor -- start 2024-03-21T08:42:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 -- start start 2024-03-21T08:42:50.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcda015bd20 0x7fcda00b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda015fba0 con 0x7fcda00cebd0 2024-03-21T08:42:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda015fe80 con 0x7fcda00b0e80 2024-03-21T08:42:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda7800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda0160160 con 0x7fcda015bd20 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda5800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcda00cebd0 0x7fcda008afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41876/0 (socket says 172.21.15.28:41876) 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda5800640 1 -- 172.21.15.28:0/4063138815 learned_addr learned my addr 172.21.15.28:0/4063138815 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda4e00640 1 --2- 172.21.15.28:0/4063138815 >> v2:172.21.15.28:3300/0 conn(0x7fcda015bd20 0x7fcda00b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980973021 0 0) 0x7fcda015fba0 con 0x7fcda00cebd0 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd84003530 con 0x7fcda00cebd0 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda4e00640 1 -- 172.21.15.28:0/4063138815 >> v1:172.21.15.151:6789/0 conn(0x7fcda00b0e80 legacy=0x7fcda00ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426918173 0 0) 0x7fcda015fe80 con 0x7fcda00b0e80 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda4e00640 1 -- 172.21.15.28:0/4063138815 >> v1:172.21.15.28:6789/0 conn(0x7fcda00cebd0 legacy=0x7fcda008afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.131+0000 7fcda4e00640 1 -- 172.21.15.28:0/4063138815 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda0160440 con 0x7fcda015bd20 2024-03-21T08:42:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda4e00640 1 --2- 172.21.15.28:0/4063138815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda015bd20 0x7fcda00b2c40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcd900097c0 tx=0x7fcd9001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8324e4cc49c0d908 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900b0070 con 0x7fcda015bd20 2024-03-21T08:42:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd900ab600 con 0x7fcda015bd20 2024-03-21T08:42:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcd9fe00640 1 -- 172.21.15.28:0/4063138815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900ab9d0 con 0x7fcda015bd20 2024-03-21T08:42:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- 172.21.15.28:0/4063138815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda015bd20 msgr2=0x7fcda00b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 --2- 172.21.15.28:0/4063138815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda015bd20 0x7fcda00b2c40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcd900097c0 tx=0x7fcd9001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- 172.21.15.28:0/4063138815 shutdown_connections 2024-03-21T08:42:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 --2- 172.21.15.28:0/4063138815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda015bd20 0x7fcda00b2c40 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- 172.21.15.28:0/4063138815 >> 172.21.15.28:0/4063138815 conn(0x7fcda00d1ab0 msgr2=0x7fcda00d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:50.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- 172.21.15.28:0/4063138815 shutdown_connections 2024-03-21T08:42:50.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- 172.21.15.28:0/4063138815 wait complete. 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 Processor -- start 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 -- start start 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.139+0000 7fcda7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 0x7fcda015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcda015bd20 0x7fcda014ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0169ea0 con 0x7fcda00b0e80 2024-03-21T08:42:50.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0169d20 con 0x7fcda015bd20 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda016a020 con 0x7fcda00cebd0 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38480/0 (socket says 172.21.15.28:38480) 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 -- 172.21.15.28:0/2929074580 learned_addr learned my addr 172.21.15.28:0/2929074580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd97e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 0x7fcda015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda5800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcda015bd20 0x7fcda014ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 -- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 msgr2=0x7fcda015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 0x7fcda015d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 -- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcda015bd20 msgr2=0x7fcda014ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcda015bd20 0x7fcda014ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 -- 172.21.15.28:0/2929074580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda006d2c0 con 0x7fcda00b0e80 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd97e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 0x7fcda015d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda4e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcd900ae040 tx=0x7fcd9001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900bb020 con 0x7fcda00b0e80 2024-03-21T08:42:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd900b1040 con 0x7fcda00b0e80 2024-03-21T08:42:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900b0050 con 0x7fcda00b0e80 2024-03-21T08:42:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda01650d0 con 0x7fcda00b0e80 2024-03-21T08:42:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda00cab00 con 0x7fcda00b0e80 2024-03-21T08:42:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda00ce5c0 con 0x7fcda00b0e80 2024-03-21T08:42:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.143+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcd900be070 con 0x7fcda00b0e80 2024-03-21T08:42:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.147+0000 7fcd9d600640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 0x7fcd7c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.147+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcd9011f890 con 0x7fcda00b0e80 2024-03-21T08:42:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.147+0000 7fcd97e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 0x7fcd7c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.147+0000 7fcd97e00640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 0x7fcd7c069fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcda0153210 tx=0x7fcd88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.151+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcd90016090 con 0x7fcda00b0e80 2024-03-21T08:42:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fcda004f4f0 con 0x7fcda00b0e80 2024-03-21T08:42:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcd9d600640 1 -- 172.21.15.28:0/2929074580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7fcd900eff10 con 0x7fcda00b0e80 2024-03-21T08:42:50.316 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 msgr2=0x7fcd7c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 0x7fcd7c069fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcda0153210 tx=0x7fcd88009040 comp rx=0 tx=0).stop 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 msgr2=0x7fcda00ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcd900ae040 tx=0x7fcd9001ace0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 shutdown_connections 2024-03-21T08:42:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcda015bd20 0x7fcda014ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcd7c067b20 0x7fcd7c069fe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcda00cebd0 0x7fcda015d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 --2- 172.21.15.28:0/2929074580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcda00b0e80 0x7fcda00ca5c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 >> 172.21.15.28:0/2929074580 conn(0x7fcda00d1ab0 msgr2=0x7fcda008afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.315+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 shutdown_connections 2024-03-21T08:42:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.319+0000 7fcda7800640 1 -- 172.21.15.28:0/2929074580 wait complete. 2024-03-21T08:42:50.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:42:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 Processor -- start 2024-03-21T08:42:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 -- start start 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 0x7f3f9815fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 0x7f3f980cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f980ce4d0 0x7f3f980b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980cd070 con 0x7f3f980ce4d0 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980b7560 con 0x7f3f9815f9e0 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f97200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f98160300 con 0x7f3f981566d0 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f980ce4d0 0x7f3f980b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f980ce4d0 0x7f3f980b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49056/0 (socket says 172.21.15.28:49056) 2024-03-21T08:42:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 -- 172.21.15.28:0/2951320654 learned_addr learned my addr 172.21.15.28:0/2951320654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f8d400640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 0x7f3f980cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 -- 172.21.15.28:0/2951320654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 msgr2=0x7f3f980cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f95e00640 1 --2- 172.21.15.28:0/2951320654 >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 0x7f3f9815fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 0x7f3f980cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 -- 172.21.15.28:0/2951320654 >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 msgr2=0x7f3f9815fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 --2- 172.21.15.28:0/2951320654 >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 0x7f3f9815fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f96800640 1 -- 172.21.15.28:0/2951320654 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f980cec70 con 0x7f3f980ce4d0 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f8d400640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 0x7f3f980cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.479+0000 7f3f95e00640 1 --2- 172.21.15.28:0/2951320654 >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 0x7f3f9815fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:42:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f96800640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f980ce4d0 0x7f3f980b0590 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3f980681c0 tx=0x7f3f9001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe04fb797a23b451 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f95400640 1 -- 172.21.15.28:0/2951320654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900a56d0 con 0x7f3f980ce4d0 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f95400640 1 -- 172.21.15.28:0/2951320654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f900a5870 con 0x7f3f980ce4d0 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f95400640 1 -- 172.21.15.28:0/2951320654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900a5c40 con 0x7f3f980ce4d0 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 -- 172.21.15.28:0/2951320654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f980ce4d0 msgr2=0x7f3f980b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f980ce4d0 0x7f3f980b0590 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3f980681c0 tx=0x7f3f9001c960 comp rx=0 tx=0).stop 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 -- 172.21.15.28:0/2951320654 shutdown_connections 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f980ce4d0 0x7f3f980b0590 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 --2- 172.21.15.28:0/2951320654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f981566d0 0x7f3f980cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 --2- 172.21.15.28:0/2951320654 >> v2:172.21.15.151:3300/0 conn(0x7f3f9815f9e0 0x7f3f9815fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 -- 172.21.15.28:0/2951320654 >> 172.21.15.28:0/2951320654 conn(0x7f3f980b3440 msgr2=0x7f3f980b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 -- 172.21.15.28:0/2951320654 shutdown_connections 2024-03-21T08:42:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.483+0000 7f3f97200640 1 -- 172.21.15.28:0/2951320654 wait complete. 2024-03-21T08:42:50.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 Processor -- start 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- start start 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f981566d0 0x7f3f980caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 0x7f3f980caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9815c3f0 con 0x7f3f9815f9e0 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f980ce920 con 0x7f3f981566d0 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9815c570 con 0x7f3f980ce4d0 2024-03-21T08:42:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f95e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 0x7f3f980caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f95e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:54844/0 (socket says 172.21.15.28:54844) 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f8d400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f981566d0 0x7f3f980caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f95e00640 1 -- 172.21.15.28:0/3386240321 learned_addr learned my addr 172.21.15.28:0/3386240321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 -- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 msgr2=0x7f3f981559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 -- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f981566d0 msgr2=0x7f3f980caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f981566d0 0x7f3f980caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 -- 172.21.15.28:0/3386240321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f980c5a10 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f95e00640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f96800640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 0x7f3f980caf50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3f980724e0 tx=0x7f3f90019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900bc030 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f900b8470 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900b8820 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f980cb590 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.487+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f980cbb50 con 0x7f3f9815f9e0 2024-03-21T08:42:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.491+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9804f410 con 0x7f3f9815f9e0 2024-03-21T08:42:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.491+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3f900bf070 con 0x7f3f9815f9e0 2024-03-21T08:42:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.491+0000 7f3f8ea00640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 0x7f3f74069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.491+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3f90120bd0 con 0x7f3f9815f9e0 2024-03-21T08:42:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.491+0000 7f3f95e00640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 0x7f3f74069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.495+0000 7f3f95e00640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 0x7f3f74069fe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.495+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3f900165e0 con 0x7f3f9815f9e0 2024-03-21T08:42:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.671+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f980712a0 con 0x7f3f9815f9e0 2024-03-21T08:42:50.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.671+0000 7f3f8ea00640 1 -- 172.21.15.28:0/3386240321 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3f900f0d20 con 0x7f3f9815f9e0 2024-03-21T08:42:50.673 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 msgr2=0x7f3f74069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 0x7f3f74069fe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88008040 comp rx=0 tx=0).stop 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 msgr2=0x7f3f980caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 0x7f3f980caf50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3f980724e0 tx=0x7f3f90019700 comp rx=0 tx=0).stop 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 shutdown_connections 2024-03-21T08:42:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f9815f9e0 0x7f3f980caf50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3f981566d0 0x7f3f980caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3f74067b20 0x7f3f74069fe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 --2- 172.21.15.28:0/3386240321 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3f980ce4d0 0x7f3f981559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 >> 172.21.15.28:0/3386240321 conn(0x7f3f980b3440 msgr2=0x7f3f980b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 shutdown_connections 2024-03-21T08:42:50.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:50.675+0000 7f3f97200640 1 -- 172.21.15.28:0/3386240321 wait complete. 2024-03-21T08:42:50.704 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:42:50.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:50.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:50.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[37535]: fuse finished with error 0 and tester_r 0 2024-03-21T08:42:50.725 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:42:56.827 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:42:56.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:56.827 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:42:56.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:42:57.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 Processor -- start 2024-03-21T08:42:57.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 -- start start 2024-03-21T08:42:57.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc0ce490 0x7f2bdc15e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2bdc0b4380 0x7f2bdc0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc15bed0 con 0x7f2bdc15ba70 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bdc15ebf0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2be1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc15eed0 con 0x7f2bdc0ce490 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bd3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2bdc15ba70 0x7f2bdc0ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35428/0 (socket says 172.21.15.28:35428) 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bd3e00640 1 -- 172.21.15.28:0/2083403845 learned_addr learned my addr 172.21.15.28:0/2083403845 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdb400640 1 --2- 172.21.15.28:0/2083403845 >> v2:172.21.15.28:3300/0 conn(0x7f2bdc0b4380 0x7f2bdc0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdaa00640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc0ce490 0x7f2bdc15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bda000640 1 -- 172.21.15.28:0/2083403845 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225357269 0 0) 0x7f2bdc15bed0 con 0x7f2bdc15ba70 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bda000640 1 -- 172.21.15.28:0/2083403845 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc0003530 con 0x7f2bdc15ba70 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdb400640 1 -- 172.21.15.28:0/2083403845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc0ce490 msgr2=0x7f2bdc15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdb400640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc0ce490 0x7f2bdc15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdb400640 1 -- 172.21.15.28:0/2083403845 >> v1:172.21.15.28:6789/0 conn(0x7f2bdc15ba70 legacy=0x7f2bdc0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.043+0000 7f2bdb400640 1 -- 172.21.15.28:0/2083403845 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc15f1b0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bdb400640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc0b47f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009d90 tx=0x7f2bd401eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620357748ab991f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bda000640 1 -- 172.21.15.28:0/2083403845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40b4070 con 0x7f2bdc0b4380 2024-03-21T08:42:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bda000640 1 -- 172.21.15.28:0/2083403845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd40a7ab0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bda000640 1 -- 172.21.15.28:0/2083403845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40af440 con 0x7f2bdc0b4380 2024-03-21T08:42:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- 172.21.15.28:0/2083403845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 msgr2=0x7f2bdc0b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc0b47f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009d90 tx=0x7f2bd401eaf0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- 172.21.15.28:0/2083403845 shutdown_connections 2024-03-21T08:42:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc0b47f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- 172.21.15.28:0/2083403845 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc0ce490 0x7f2bdc15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- 172.21.15.28:0/2083403845 >> 172.21.15.28:0/2083403845 conn(0x7f2bdc0cc230 msgr2=0x7f2bdc0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- 172.21.15.28:0/2083403845 shutdown_connections 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- 172.21.15.28:0/2083403845 wait complete. 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 Processor -- start 2024-03-21T08:42:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- start start 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bdc0ce490 0x7f2bdc0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc15ba70 0x7f2bdc14a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc0cd500 con 0x7f2bdc0b4380 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc0cd380 con 0x7f2bdc0ce490 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2be1a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc0cd680 con 0x7f2bdc15ba70 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bdaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bdaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49074/0 (socket says 172.21.15.28:49074) 2024-03-21T08:42:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.051+0000 7f2bdaa00640 1 -- 172.21.15.28:0/16198134 learned_addr learned my addr 172.21.15.28:0/16198134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd3e00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bdc0ce490 0x7f2bdc0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdb400640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc15ba70 0x7f2bdc14a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 -- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc15ba70 msgr2=0x7f2bdc14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc15ba70 0x7f2bdc14a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 -- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bdc0ce490 msgr2=0x7f2bdc0adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bdc0ce490 0x7f2bdc0adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 -- 172.21.15.28:0/16198134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc0acaf0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bdaa00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00e930 tx=0x7f2bcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc017070 con 0x7f2bdc0b4380 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bcc00ca00 con 0x7f2bdc0b4380 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00cdb0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc0acdd0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc0ad390 con 0x7f2bdc0b4380 2024-03-21T08:42:57.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc04f410 con 0x7f2bdc0b4380 2024-03-21T08:42:57.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2bcc01a070 con 0x7f2bdc0b4380 2024-03-21T08:42:57.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.055+0000 7f2bd2a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 0x7f2bb8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:57.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.059+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2bcc0866b0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.059+0000 7f2bd3e00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 0x7f2bb8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:57.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.059+0000 7f2bd3e00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 0x7f2bb8069fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc15ba00 tx=0x7f2bc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.063+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2bcc056df0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.259+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2bdc072c10 con 0x7f2bdc0b4380 2024-03-21T08:42:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2bd2a00640 1 -- 172.21.15.28:0/16198134 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+82 (secure 0 0 0) 0x7f2bcc0566f0 con 0x7f2bdc0b4380 2024-03-21T08:42:57.264 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 msgr2=0x7f2bb8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 0x7f2bb8069fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc15ba00 tx=0x7f2bc4009040 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 msgr2=0x7f2bdc1492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00e930 tx=0x7f2bcc00edf0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 shutdown_connections 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2bb8067b20 0x7f2bb8069fe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2bdc15ba70 0x7f2bdc14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2bdc0ce490 0x7f2bdc0adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 --2- 172.21.15.28:0/16198134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc0b4380 0x7f2bdc1492d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 >> 172.21.15.28:0/16198134 conn(0x7f2bdc0cc230 msgr2=0x7f2bdc150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 shutdown_connections 2024-03-21T08:42:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:57.263+0000 7f2be1a00640 1 -- 172.21.15.28:0/16198134 wait complete. 2024-03-21T08:42:57.285 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:42:57.286 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:42:57.286 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:42:57.286 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:42:57.286 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:57.287 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:42:57.287 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:42:57.287 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:42:57.287 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:57.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.288 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:42:57.297 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:42:57.298 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42292sec preferred_lft 42292sec 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:42:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:57.300 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:42:57.301 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:42:57.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.301 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:42:57.360 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:42:57.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.362 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:42:57.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:42:57.413 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:42:57.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.504 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:42:57.513 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:42:57.513 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:42:57.513 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:57.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:42:57.574 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:42:57.575 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:42:57.575 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:57.627 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:42:57.627 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:42:57.627 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:57.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:42:57.740 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:42:57.793 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:42:57.794 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:57.795 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:57.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.796 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:42:57.848 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:42:57.848 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:42:57.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:42:57.921 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:42:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:42:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:42:57.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:57.978 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:42:57.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:42:57.991+0000 7ff775f1e580 -1 init, newargv = 0x5639ffa527a0 newargc=15 2024-03-21T08:42:57.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38094]: starting ceph client 2024-03-21T08:42:58.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38094]: starting fuse 2024-03-21T08:42:58.006 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:42:58.007 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:42:58.007 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:42:58.008 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:42:58.009 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:42:58.009 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:42:58.009 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:42:58.009 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:42:58.009 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:42:58.138 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.38094.asok 2024-03-21T08:42:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:42:58.150 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38094.asok 2024-03-21T08:42:58.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:58.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38094.asok status 2024-03-21T08:42:58.323 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:42:58.323 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "38094", 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5378, 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:42:58.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5378 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 558766365 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 558766365 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:42:58.325 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.5378 172.21.15.28:0/558766365", 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "172.21.15.28:0/558766365", 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 19, 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 42, 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:42:58.326 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:42:58.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:58.339 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:42:58.339 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:58.349 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:42:58.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:58.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:58.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:42:58.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:58.411 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-21T08:42:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:setfattr: layoutfile: Permission denied 2024-03-21T08:42:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:42:58.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:42:58.466 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-21T08:42:58.516 INFO:teuthology.orchestra.run.smithi028.stderr:setfattr: layoutdir: Permission denied 2024-03-21T08:42:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:42:58.517 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:42:58.518 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:42:58.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:42:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 Processor -- start 2024-03-21T08:42:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 -- start start 2024-03-21T08:42:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00af060 0x7f99e00cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99e00b34f0 0x7f99e00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f99e00b5350 0x7f99e0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e00ce6e0 con 0x7f99e00b34f0 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e015bf00 con 0x7f99e00b5350 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99e5800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00cdad0 con 0x7f99e00af060 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99e00b34f0 0x7f99e00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99e00b34f0 0x7f99e00b4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49076/0 (socket says 172.21.15.28:49076) 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 -- 172.21.15.28:0/3424847641 learned_addr learned my addr 172.21.15.28:0/3424847641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99df400640 1 --2- 172.21.15.28:0/3424847641 >> v2:172.21.15.151:3300/0 conn(0x7f99e00b5350 0x7f99e0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99dea00640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00af060 0x7f99e00cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 -- 172.21.15.28:0/3424847641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00af060 msgr2=0x7f99e00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00af060 0x7f99e00cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 -- 172.21.15.28:0/3424847641 >> v2:172.21.15.151:3300/0 conn(0x7f99e00b5350 msgr2=0x7f99e0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 --2- 172.21.15.28:0/3424847641 >> v2:172.21.15.151:3300/0 conn(0x7f99e00b5350 0x7f99e0071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.751+0000 7f99de000640 1 -- 172.21.15.28:0/3424847641 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e00cddb0 con 0x7f99e00b34f0 2024-03-21T08:42:58.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99de000640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e00b4e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f99e00cc140 tx=0x7f99cc01c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ce4e8741d3c7ad server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:58.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99dd600640 1 -- 172.21.15.28:0/3424847641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0b0070 con 0x7f99e00b34f0 2024-03-21T08:42:58.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99dd600640 1 -- 172.21.15.28:0/3424847641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc0ab600 con 0x7f99e00b34f0 2024-03-21T08:42:58.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99dd600640 1 -- 172.21.15.28:0/3424847641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0ab9d0 con 0x7f99e00b34f0 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 -- 172.21.15.28:0/3424847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 msgr2=0x7f99e00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e00b4e10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f99e00cc140 tx=0x7f99cc01c1d0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 -- 172.21.15.28:0/3424847641 shutdown_connections 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 --2- 172.21.15.28:0/3424847641 >> v2:172.21.15.151:3300/0 conn(0x7f99e00b5350 0x7f99e0071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e00b4e10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 --2- 172.21.15.28:0/3424847641 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00af060 0x7f99e00cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 -- 172.21.15.28:0/3424847641 >> 172.21.15.28:0/3424847641 conn(0x7f99e00cc230 msgr2=0x7f99e00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 -- 172.21.15.28:0/3424847641 shutdown_connections 2024-03-21T08:42:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.755+0000 7f99e5800640 1 -- 172.21.15.28:0/3424847641 wait complete. 2024-03-21T08:42:58.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 Processor -- start 2024-03-21T08:42:58.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 -- start start 2024-03-21T08:42:58.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00b5350 0x7f99e00ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 0x7f99e015d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00b5730 con 0x7f99e00b34f0 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e015c200 con 0x7f99e00ade90 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99e5800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e006da60 con 0x7f99e00b5350 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49084/0 (socket says 172.21.15.28:49084) 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 -- 172.21.15.28:0/3853473392 learned_addr learned my addr 172.21.15.28:0/3853473392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99de000640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00b5350 0x7f99e00ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99df400640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 0x7f99e015d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 -- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00b5350 msgr2=0x7f99e00ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00b5350 0x7f99e00ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 -- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 msgr2=0x7f99e015d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 0x7f99e015d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 -- 172.21.15.28:0/3853473392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e00ae370 con 0x7f99e00b34f0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99df400640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 0x7f99e015d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.759+0000 7f99dea00640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f99d000d800 tx=0x7f99d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0018020 con 0x7f99e00b34f0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d00096a0 con 0x7f99e00b34f0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e00ae650 con 0x7f99e00b34f0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0009a50 con 0x7f99e00b34f0 2024-03-21T08:42:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e01f6a00 con 0x7f99e00b34f0 2024-03-21T08:42:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e004f410 con 0x7f99e00b34f0 2024-03-21T08:42:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f99d001b020 con 0x7f99e00b34f0 2024-03-21T08:42:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 0x7f99c0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.763+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99d00867a0 con 0x7f99e00b34f0 2024-03-21T08:42:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.767+0000 7f99de000640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 0x7f99c0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.767+0000 7f99de000640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 0x7f99c0069fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0ad000 tx=0x7f99cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.767+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f99d0022020 con 0x7f99e00b34f0 2024-03-21T08:42:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.935+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99e00ccd50 con 0x7f99e00b34f0 2024-03-21T08:42:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99d6a00640 1 -- 172.21.15.28:0/3853473392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f99d0056e20 con 0x7f99e00b34f0 2024-03-21T08:42:58.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:42:58.940 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:42:58.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 msgr2=0x7f99c0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:58.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 0x7f99c0069fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99cc0ad000 tx=0x7f99cc008040 comp rx=0 tx=0).stop 2024-03-21T08:42:58.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 msgr2=0x7f99e013e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:58.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f99d000d800 tx=0x7f99d000dcc0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 shutdown_connections 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99e00ade90 0x7f99e015d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f99c0067b20 0x7f99c0069fe0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99e00b5350 0x7f99e00ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 --2- 172.21.15.28:0/3853473392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e00b34f0 0x7f99e013e2b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 >> 172.21.15.28:0/3853473392 conn(0x7f99e00cc230 msgr2=0x7f99e015ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 shutdown_connections 2024-03-21T08:42:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:58.939+0000 7f99e5800640 1 -- 172.21.15.28:0/3853473392 wait complete. 2024-03-21T08:42:58.961 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:42:58.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:42:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 Processor -- start 2024-03-21T08:42:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 -- start start 2024-03-21T08:42:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29a006b880 0x7f29a00ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a00cfbb0 0x7f29a00cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a00cebc0 con 0x7f29a0152b60 2024-03-21T08:42:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a00ceea0 con 0x7f29a006b880 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f29a7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a00cf180 con 0x7f29a00cfbb0 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.103+0000 7f299fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29a006b880 0x7f29a00ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29a006b880 0x7f29a00ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49086/0 (socket says 172.21.15.28:49086) 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 -- 172.21.15.28:0/2189657772 learned_addr learned my addr 172.21.15.28:0/2189657772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f29a5800640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a00cfbb0 0x7f29a00cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 -- 172.21.15.28:0/2189657772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a00cfbb0 msgr2=0x7f29a00cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a00cfbb0 0x7f29a00cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 -- 172.21.15.28:0/2189657772 >> v1:172.21.15.151:6789/0 conn(0x7f29a0152b60 legacy=0x7f29a006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 -- 172.21.15.28:0/2189657772 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a00cf460 con 0x7f29a006b880 2024-03-21T08:42:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299fe00640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a00ce520 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f298c009870 tx=0x7f298c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eacf964e1a02de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299f400640 1 -- 172.21.15.28:0/2189657772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b2070 con 0x7f29a006b880 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299f400640 1 -- 172.21.15.28:0/2189657772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c0a9070 con 0x7f29a006b880 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.107+0000 7f299f400640 1 -- 172.21.15.28:0/2189657772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c0ad880 con 0x7f29a006b880 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- 172.21.15.28:0/2189657772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 msgr2=0x7f29a00ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a00ce520 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f298c009870 tx=0x7f298c01c560 comp rx=0 tx=0).stop 2024-03-21T08:42:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- 172.21.15.28:0/2189657772 shutdown_connections 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a00cfbb0 0x7f29a00cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- 172.21.15.28:0/2189657772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a00ce520 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- 172.21.15.28:0/2189657772 >> 172.21.15.28:0/2189657772 conn(0x7f29a0099a30 msgr2=0x7f29a0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- 172.21.15.28:0/2189657772 shutdown_connections 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- 172.21.15.28:0/2189657772 wait complete. 2024-03-21T08:42:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 Processor -- start 2024-03-21T08:42:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- start start 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29a00cfbb0 0x7f29a0147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a0152b60 0x7f29a01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a015c9c0 con 0x7f29a006b880 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a015c840 con 0x7f29a00cfbb0 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a7800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a015cb40 con 0x7f29a0152b60 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49098/0 (socket says 172.21.15.28:49098) 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 -- 172.21.15.28:0/4289083337 learned_addr learned my addr 172.21.15.28:0/4289083337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a5800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a0152b60 0x7f29a01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f299fe00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29a00cfbb0 0x7f29a0147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 -- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a0152b60 msgr2=0x7f29a01551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a0152b60 0x7f29a01551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 -- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29a00cfbb0 msgr2=0x7f29a0147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29a00cfbb0 0x7f29a0147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.111+0000 7f29a4e00640 1 -- 172.21.15.28:0/4289083337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a014fcb0 con 0x7f29a006b880 2024-03-21T08:42:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f29a4e00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f299400c370 tx=0x7f299400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29940052d0 con 0x7f29a006b880 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994005470 con 0x7f29a006b880 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a014ff90 con 0x7f29a006b880 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994007260 con 0x7f29a006b880 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a0150550 con 0x7f29a006b880 2024-03-21T08:42:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a009ea60 con 0x7f29a006b880 2024-03-21T08:42:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.115+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2994007400 con 0x7f29a006b880 2024-03-21T08:42:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.119+0000 7f299cc00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 0x7f298006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.119+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f29940855a0 con 0x7f29a006b880 2024-03-21T08:42:59.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.119+0000 7f299fe00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 0x7f298006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.119+0000 7f299fe00640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 0x7f298006a010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f298c0af000 tx=0x7f298c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.123+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2994055660 con 0x7f29a006b880 2024-03-21T08:42:59.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.287+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29a004f4f0 con 0x7f29a006b880 2024-03-21T08:42:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.287+0000 7f299cc00640 1 -- 172.21.15.28:0/4289083337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2994059700 con 0x7f29a006b880 2024-03-21T08:42:59.290 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 msgr2=0x7f298006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 0x7f298006a010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f298c0af000 tx=0x7f298c008040 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 msgr2=0x7f29a0150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f299400c370 tx=0x7f299400c830 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 shutdown_connections 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2980067b50 0x7f298006a010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29a0152b60 0x7f29a01551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29a00cfbb0 0x7f29a0147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 --2- 172.21.15.28:0/4289083337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006b880 0x7f29a0150d50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 >> 172.21.15.28:0/4289083337 conn(0x7f29a0099a30 msgr2=0x7f29a014ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 shutdown_connections 2024-03-21T08:42:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.291+0000 7f29a7800640 1 -- 172.21.15.28:0/4289083337 wait complete. 2024-03-21T08:42:59.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:42:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 Processor -- start 2024-03-21T08:42:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 -- start start 2024-03-21T08:42:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 0x7fb6d808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6d81626d0 0x7fb6d80ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d808b8f0 con 0x7fb6d815dd40 2024-03-21T08:42:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d806c280 con 0x7fb6d81626d0 2024-03-21T08:42:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6dfc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d808bbd0 con 0x7fb6d815e120 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6d81626d0 0x7fb6d80ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6d81626d0 0x7fb6d80ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49110/0 (socket says 172.21.15.28:49110) 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 -- 172.21.15.28:0/3196628236 learned_addr learned my addr 172.21.15.28:0/3196628236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6d7e00640 1 --2- 172.21.15.28:0/3196628236 >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 0x7fb6d808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 -- 172.21.15.28:0/3196628236 >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 msgr2=0x7fb6d808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 --2- 172.21.15.28:0/3196628236 >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 0x7fb6d808afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 -- 172.21.15.28:0/3196628236 >> v1:172.21.15.151:6789/0 conn(0x7fb6d815dd40 legacy=0x7fb6d80cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6ddc00640 1 -- 172.21.15.28:0/3196628236 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d808beb0 con 0x7fb6d81626d0 2024-03-21T08:42:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.455+0000 7fb6d7e00640 1 --2- 172.21.15.28:0/3196628236 >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 0x7fb6d808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:42:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.459+0000 7fb6ddc00640 1 --2- 172.21.15.28:0/3196628236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d81626d0 0x7fb6d80ce3a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80d28a0 tx=0x7fb6d001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4268b77cc9e1cf server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.459+0000 7fb6d7400640 1 -- 172.21.15.28:0/3196628236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d00a5620 con 0x7fb6d81626d0 2024-03-21T08:42:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.459+0000 7fb6d7400640 1 -- 172.21.15.28:0/3196628236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d00a57c0 con 0x7fb6d81626d0 2024-03-21T08:42:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.459+0000 7fb6d7400640 1 -- 172.21.15.28:0/3196628236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d00a5b90 con 0x7fb6d81626d0 2024-03-21T08:42:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3196628236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d81626d0 msgr2=0x7fb6d80ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3196628236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d81626d0 0x7fb6d80ce3a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80d28a0 tx=0x7fb6d001c960 comp rx=0 tx=0).stop 2024-03-21T08:42:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3196628236 shutdown_connections 2024-03-21T08:42:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3196628236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d81626d0 0x7fb6d80ce3a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3196628236 >> v2:172.21.15.151:3300/0 conn(0x7fb6d815e120 0x7fb6d808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3196628236 >> 172.21.15.28:0/3196628236 conn(0x7fb6d80d1b20 msgr2=0x7fb6d80d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3196628236 shutdown_connections 2024-03-21T08:42:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3196628236 wait complete. 2024-03-21T08:42:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 Processor -- start 2024-03-21T08:42:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- start start 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6d815e120 0x7fb6d80b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb6d81626d0 0x7fb6d80b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d806d590 con 0x7fb6d815dd40 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d806d410 con 0x7fb6d815e120 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dfc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80cecf0 con 0x7fb6d81626d0 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49114/0 (socket says 172.21.15.28:49114) 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 -- 172.21.15.28:0/3483317706 learned_addr learned my addr 172.21.15.28:0/3483317706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6d7e00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6d815e120 0x7fb6d80b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6ddc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb6d81626d0 0x7fb6d80b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 -- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb6d81626d0 msgr2=0x7fb6d80b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb6d81626d0 0x7fb6d80b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 -- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6d815e120 msgr2=0x7fb6d80b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6d815e120 0x7fb6d80b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.463+0000 7fb6dd200640 1 -- 172.21.15.28:0/3483317706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80c6100 con 0x7fb6d815dd40 2024-03-21T08:42:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6dd200640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c3f0 tx=0x7fb6cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc017070 con 0x7fb6d815dd40 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc0052d0 con 0x7fb6d815dd40 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d80c63e0 con 0x7fb6d815dd40 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc003840 con 0x7fb6d815dd40 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d80c69a0 con 0x7fb6d815dd40 2024-03-21T08:42:59.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d809fb90 con 0x7fb6d815dd40 2024-03-21T08:42:59.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.467+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb6cc01a070 con 0x7fb6d815dd40 2024-03-21T08:42:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.471+0000 7fb6d4c00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 0x7fb6b8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.471+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb6cc085970 con 0x7fb6d815dd40 2024-03-21T08:42:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.471+0000 7fb6d7e00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 0x7fb6b8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.471+0000 7fb6d7e00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 0x7fb6b8069fe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8168c60 tx=0x7fb6c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.471+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb6cc021020 con 0x7fb6d815dd40 2024-03-21T08:42:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.639+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb6d80c5370 con 0x7fb6d815dd40 2024-03-21T08:42:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.639+0000 7fb6d4c00640 1 -- 172.21.15.28:0/3483317706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb6cc005480 con 0x7fb6d815dd40 2024-03-21T08:42:59.643 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 msgr2=0x7fb6b8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 0x7fb6b8069fe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8168c60 tx=0x7fb6c4008040 comp rx=0 tx=0).stop 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 msgr2=0x7fb6d80c71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c3f0 tx=0x7fb6cc00c8b0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 shutdown_connections 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb6b8067b20 0x7fb6b8069fe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb6d81626d0 0x7fb6d80b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb6d815e120 0x7fb6d80b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 --2- 172.21.15.28:0/3483317706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6d815dd40 0x7fb6d80c71a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 >> 172.21.15.28:0/3483317706 conn(0x7fb6d80d1b20 msgr2=0x7fb6d80afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 shutdown_connections 2024-03-21T08:42:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.643+0000 7fb6dfc00640 1 -- 172.21.15.28:0/3483317706 wait complete. 2024-03-21T08:42:59.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:42:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.807+0000 7fd47e800640 1 Processor -- start 2024-03-21T08:42:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.807+0000 7fd47e800640 1 -- start start 2024-03-21T08:42:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd47e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd4780cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd47e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780b3010 con 0x7fd4780d0190 2024-03-21T08:42:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd47e800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47806d310 con 0x7fd47806cdb0 2024-03-21T08:42:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd47e800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780d05f0 con 0x7fd47815e1e0 2024-03-21T08:42:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd46ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd4780d0190 0x7fd47806c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35436/0 (socket says 172.21.15.28:35436) 2024-03-21T08:42:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd46ea00640 1 -- 172.21.15.28:0/2430962191 learned_addr learned my addr 172.21.15.28:0/2430962191 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477400640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd4780cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219063167 0 0) 0x7fd4780b3010 con 0x7fd4780d0190 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd45c003530 con 0x7fd4780d0190 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600923084 0 0) 0x7fd47806d310 con 0x7fd47806cdb0 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4780b3010 con 0x7fd47806cdb0 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2988535914 0 0) 0x7fd45c003530 con 0x7fd4780d0190 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 msgr2=0x7fd4780cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd4780cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 >> v1:172.21.15.151:6789/0 conn(0x7fd47806cdb0 legacy=0x7fd4780cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47806d5f0 con 0x7fd4780d0190 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fd460001a00 con 0x7fd4780d0190 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477400640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd4780cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 >> v1:172.21.15.28:6789/0 conn(0x7fd4780d0190 legacy=0x7fd47806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd45c003b80 0x7fd45c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd45c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c004d50 0x7fd45c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c00ba80 con 0x7fd45c004d50 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c00bc00 con 0x7fd45c003b80 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c00bd80 con 0x7fd47815e1e0 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c004d50 0x7fd45c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477400640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd45c003b80 0x7fd45c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd46ea00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd45c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 -- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 msgr2=0x7fd45c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd47815e1e0 0x7fd45c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 -- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd45c003b80 msgr2=0x7fd45c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd45c003b80 0x7fd45c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477e00640 1 -- 172.21.15.28:0/2430962191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c003530 con 0x7fd45c004d50 2024-03-21T08:42:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.811+0000 7fd477400640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd45c003b80 0x7fd45c003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:42:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd477e00640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c004d50 0x7fd45c0051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd4700018d0 tx=0x7fd47001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2790bab17f5f77b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470001520 con 0x7fd45c004d50 2024-03-21T08:42:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4700052f0 con 0x7fd45c004d50 2024-03-21T08:42:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd476a00640 1 -- 172.21.15.28:0/2430962191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470003840 con 0x7fd45c004d50 2024-03-21T08:42:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- 172.21.15.28:0/2430962191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c004d50 msgr2=0x7fd45c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 --2- 172.21.15.28:0/2430962191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c004d50 0x7fd45c0051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd4700018d0 tx=0x7fd47001c540 comp rx=0 tx=0).stop 2024-03-21T08:42:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd477e00640 1 -- 172.21.15.28:0/2430962191 reap_dead start 2024-03-21T08:42:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- 172.21.15.28:0/2430962191 shutdown_connections 2024-03-21T08:42:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- 172.21.15.28:0/2430962191 >> 172.21.15.28:0/2430962191 conn(0x7fd4780d1ab0 msgr2=0x7fd4780d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- 172.21.15.28:0/2430962191 shutdown_connections 2024-03-21T08:42:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- 172.21.15.28:0/2430962191 wait complete. 2024-03-21T08:42:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 Processor -- start 2024-03-21T08:42:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.815+0000 7fd47e800640 1 -- start start 2024-03-21T08:42:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd45c004d50 0x7fd478157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd478155930 0x7fd478155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780cfc50 con 0x7fd478160580 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47806dbb0 con 0x7fd478155930 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780cfdd0 con 0x7fd45c004d50 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49128/0 (socket says 172.21.15.28:49128) 2024-03-21T08:42:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 -- 172.21.15.28:0/3341271175 learned_addr learned my addr 172.21.15.28:0/3341271175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd477e00640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd478155930 0x7fd478155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd477400640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd45c004d50 0x7fd478157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 -- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd45c004d50 msgr2=0x7fd478157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd45c004d50 0x7fd478157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 -- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd478155930 msgr2=0x7fd478155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd478155930 0x7fd478155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 -- 172.21.15.28:0/3341271175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478157bb0 con 0x7fd478160580 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46ea00640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd460006fd0 tx=0x7fd4600115c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46001c070 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460004d90 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4780b5210 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460010040 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4780b57d0 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.819+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47804f4f0 con 0x7fd478160580 2024-03-21T08:42:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.823+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd46001f070 con 0x7fd478160580 2024-03-21T08:42:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.823+0000 7fd46fe00640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:42:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.823+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd460089380 con 0x7fd478160580 2024-03-21T08:42:59.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.823+0000 7fd477400640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:42:59.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.823+0000 7fd477400640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd468009d70 tx=0x7fd468009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:42:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.827+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd460059440 con 0x7fd478160580 2024-03-21T08:42:59.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.991+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4780d2830 con 0x7fd478160580 2024-03-21T08:42:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.991+0000 7fd46fe00640 1 -- 172.21.15.28:0/3341271175 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd46005d4e0 con 0x7fd478160580 2024-03-21T08:42:59.994 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:42:59.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 msgr2=0x7fd45406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd468009d70 tx=0x7fd468009190 comp rx=0 tx=0).stop 2024-03-21T08:42:59.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 msgr2=0x7fd478160a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:42:59.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd460006fd0 tx=0x7fd4600115c0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 shutdown_connections 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd478155930 0x7fd478155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd478160580 0x7fd478160a00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd454067b50 0x7fd45406a010 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 --2- 172.21.15.28:0/3341271175 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd45c004d50 0x7fd478157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 >> 172.21.15.28:0/3341271175 conn(0x7fd4780d1ab0 msgr2=0x7fd4780cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 shutdown_connections 2024-03-21T08:42:59.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:42:59.995+0000 7fd47e800640 1 -- 172.21.15.28:0/3341271175 wait complete. 2024-03-21T08:43:00.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:43:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.155+0000 7fb963800640 1 Processor -- start 2024-03-21T08:43:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 -- start start 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c06f0a0 0x7fb95c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c06f500 con 0x7fb95c071a80 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c0ce380 con 0x7fb95c15cbb0 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb963800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c15c720 con 0x7fb95c06f0a0 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49132/0 (socket says 172.21.15.28:49132) 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 -- 172.21.15.28:0/1007035309 learned_addr learned my addr 172.21.15.28:0/1007035309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb95be00640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c06f0a0 0x7fb95c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 -- 172.21.15.28:0/1007035309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c06f0a0 msgr2=0x7fb95c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c06f0a0 0x7fb95c0cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 -- 172.21.15.28:0/1007035309 >> v1:172.21.15.151:6789/0 conn(0x7fb95c071a80 legacy=0x7fb95c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.159+0000 7fb961800640 1 -- 172.21.15.28:0/1007035309 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c0ce5b0 con 0x7fb95c15cbb0 2024-03-21T08:43:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb961800640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb954009e50 tx=0x7fb95401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b881ada4b15a6b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb95b400640 1 -- 172.21.15.28:0/1007035309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9540a5830 con 0x7fb95c15cbb0 2024-03-21T08:43:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb95b400640 1 -- 172.21.15.28:0/1007035309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9540a59d0 con 0x7fb95c15cbb0 2024-03-21T08:43:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb95b400640 1 -- 172.21.15.28:0/1007035309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9540a5da0 con 0x7fb95c15cbb0 2024-03-21T08:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 -- 172.21.15.28:0/1007035309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c15cbb0 msgr2=0x7fb95c0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb954009e50 tx=0x7fb95401c970 comp rx=0 tx=0).stop 2024-03-21T08:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 -- 172.21.15.28:0/1007035309 shutdown_connections 2024-03-21T08:43:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c15cbb0 0x7fb95c0cc4b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 --2- 172.21.15.28:0/1007035309 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c06f0a0 0x7fb95c0cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 -- 172.21.15.28:0/1007035309 >> 172.21.15.28:0/1007035309 conn(0x7fb95c0674f0 msgr2=0x7fb95c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 -- 172.21.15.28:0/1007035309 shutdown_connections 2024-03-21T08:43:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 -- 172.21.15.28:0/1007035309 wait complete. 2024-03-21T08:43:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.163+0000 7fb963800640 1 Processor -- start 2024-03-21T08:43:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- start start 2024-03-21T08:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb95c071a80 0x7fb95c140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c15cbb0 0x7fb95c0b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c144b20 con 0x7fb95c06f0a0 2024-03-21T08:43:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0cce00 con 0x7fb95c071a80 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c168f20 con 0x7fb95c15cbb0 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49140/0 (socket says 172.21.15.28:49140) 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 -- 172.21.15.28:0/1781300297 learned_addr learned my addr 172.21.15.28:0/1781300297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb961800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c15cbb0 0x7fb95c0b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb95be00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb95c071a80 0x7fb95c140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 -- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c15cbb0 msgr2=0x7fb95c0b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c15cbb0 0x7fb95c0b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 -- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb95c071a80 msgr2=0x7fb95c140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb95c071a80 0x7fb95c140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 -- 172.21.15.28:0/1781300297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c158940 con 0x7fb95c06f0a0 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb960e00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c420 tx=0x7fb95000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9500052d0 con 0x7fb95c06f0a0 2024-03-21T08:43:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb950005470 con 0x7fb95c06f0a0 2024-03-21T08:43:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c158c20 con 0x7fb95c06f0a0 2024-03-21T08:43:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950007260 con 0x7fb95c06f0a0 2024-03-21T08:43:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c1591e0 con 0x7fb95c06f0a0 2024-03-21T08:43:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.167+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c073430 con 0x7fb95c06f0a0 2024-03-21T08:43:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.171+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb950007400 con 0x7fb95c06f0a0 2024-03-21T08:43:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.171+0000 7fb958c00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 0x7fb93c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.171+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb950085660 con 0x7fb95c06f0a0 2024-03-21T08:43:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.171+0000 7fb95be00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 0x7fb93c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.171+0000 7fb95be00640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 0x7fb93c069fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb95c143ad0 tx=0x7fb948008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.179+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb950055720 con 0x7fb95c06f0a0 2024-03-21T08:43:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.343+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fb95c072d30 con 0x7fb95c06f0a0 2024-03-21T08:43:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.343+0000 7fb958c00640 1 -- 172.21.15.28:0/1781300297 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7fb9500597c0 con 0x7fb95c06f0a0 2024-03-21T08:43:00.345 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 msgr2=0x7fb93c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 0x7fb93c069fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb95c143ad0 tx=0x7fb948008040 comp rx=0 tx=0).stop 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 msgr2=0x7fb95c1582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c420 tx=0x7fb95000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 shutdown_connections 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb93c067b20 0x7fb93c069fe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb95c15cbb0 0x7fb95c0b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb95c071a80 0x7fb95c140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 --2- 172.21.15.28:0/1781300297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95c06f0a0 0x7fb95c1582d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 >> 172.21.15.28:0/1781300297 conn(0x7fb95c0674f0 msgr2=0x7fb95c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 shutdown_connections 2024-03-21T08:43:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.347+0000 7fb963800640 1 -- 172.21.15.28:0/1781300297 wait complete. 2024-03-21T08:43:00.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 Processor -- start 2024-03-21T08:43:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 -- start start 2024-03-21T08:43:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee015b970 0x7faee0070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee015bd50 con 0x7faee006b170 2024-03-21T08:43:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee01602d0 con 0x7faee00d0870 2024-03-21T08:43:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00ce370 con 0x7faee015b970 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee015b970 0x7faee0070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee015b970 0x7faee0070b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47354/0 (socket says 172.21.15.28:47354) 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 -- 172.21.15.28:0/2260505869 learned_addr learned my addr 172.21.15.28:0/2260505869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 -- 172.21.15.28:0/2260505869 >> v1:172.21.15.151:6789/0 conn(0x7faee00d0870 legacy=0x7faee015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 -- 172.21.15.28:0/2260505869 >> v1:172.21.15.28:6789/0 conn(0x7faee006b170 legacy=0x7faee015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.511+0000 7faedde00640 1 -- 172.21.15.28:0/2260505869 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee00cdbb0 con 0x7faee015b970 2024-03-21T08:43:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedde00640 1 --2- 172.21.15.28:0/2260505869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee015b970 0x7faee0070b90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a44b48fe0673c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedd400640 1 -- 172.21.15.28:0/2260505869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00b0070 con 0x7faee015b970 2024-03-21T08:43:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedd400640 1 -- 172.21.15.28:0/2260505869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed00ab600 con 0x7faee015b970 2024-03-21T08:43:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedd400640 1 -- 172.21.15.28:0/2260505869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00ab9d0 con 0x7faee015b970 2024-03-21T08:43:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 -- 172.21.15.28:0/2260505869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee015b970 msgr2=0x7faee0070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 --2- 172.21.15.28:0/2260505869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee015b970 0x7faee0070b90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faed0009870 tx=0x7faed001c520 comp rx=0 tx=0).stop 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 -- 172.21.15.28:0/2260505869 shutdown_connections 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 --2- 172.21.15.28:0/2260505869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee015b970 0x7faee0070b90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 -- 172.21.15.28:0/2260505869 >> 172.21.15.28:0/2260505869 conn(0x7faee00b23d0 msgr2=0x7faee00b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 -- 172.21.15.28:0/2260505869 shutdown_connections 2024-03-21T08:43:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.515+0000 7faedf200640 1 -- 172.21.15.28:0/2260505869 wait complete. 2024-03-21T08:43:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 Processor -- start 2024-03-21T08:43:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- start start 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faee006b170 0x7faee0147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faee015b970 0x7faee0151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee006b990 con 0x7faee00d0870 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee00737c0 con 0x7faee006b170 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee0153830 con 0x7faee015b970 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47366/0 (socket says 172.21.15.28:47366) 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 -- 172.21.15.28:0/387129713 learned_addr learned my addr 172.21.15.28:0/387129713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedde00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faee006b170 0x7faee0147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faede800640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faee015b970 0x7faee0151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 -- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faee015b970 msgr2=0x7faee0151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faee015b970 0x7faee0151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 -- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faee006b170 msgr2=0x7faee0147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faee006b170 0x7faee0147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 -- 172.21.15.28:0/387129713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee015edb0 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed7e00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faec800c420 tx=0x7faec800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80052d0 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8005470 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee0152f00 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8007260 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee01534c0 con 0x7faee00d0870 2024-03-21T08:43:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.519+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee004f410 con 0x7faee00d0870 2024-03-21T08:43:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.523+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faec8007400 con 0x7faee00d0870 2024-03-21T08:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.523+0000 7faed6000640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 0x7faebc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.523+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faec80855a0 con 0x7faee00d0870 2024-03-21T08:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.523+0000 7faedde00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 0x7faebc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.523+0000 7faedde00640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 0x7faebc069fe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faed00ad000 tx=0x7faed00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.527+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faee015bd50 con 0x7faee00d0870 2024-03-21T08:43:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faee0152890 con 0x7faee00d0870 2024-03-21T08:43:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faed6000640 1 -- 172.21.15.28:0/387129713 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7faec80556e0 con 0x7faee00d0870 2024-03-21T08:43:00.705 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 msgr2=0x7faebc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 0x7faebc069fe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faed00ad000 tx=0x7faed00a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 msgr2=0x7faee0150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faec800c420 tx=0x7faec800c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 shutdown_connections 2024-03-21T08:43:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faebc067b20 0x7faebc069fe0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faee015b970 0x7faee0151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee00d0870 0x7faee0150ed0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 --2- 172.21.15.28:0/387129713 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faee006b170 0x7faee0147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 >> 172.21.15.28:0/387129713 conn(0x7faee00b23d0 msgr2=0x7faee0155b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 shutdown_connections 2024-03-21T08:43:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.703+0000 7faedf200640 1 -- 172.21.15.28:0/387129713 wait complete. 2024-03-21T08:43:00.728 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:00.728 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:00.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 Processor -- start 2024-03-21T08:43:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 -- start start 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe36c15e0c0 0x7fe36c06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 0x7fe36c099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0cc1d0 con 0x7fe36c0ccc70 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0d1730 con 0x7fe36c15e0c0 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe373a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c09a520 con 0x7fe36c099b80 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe371000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe36c15e0c0 0x7fe36c06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe371000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe36c15e0c0 0x7fe36c06cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47372/0 (socket says 172.21.15.28:47372) 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe371a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe36c0ccc70 0x7fe36c0cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46598/0 (socket says 172.21.15.28:46598) 2024-03-21T08:43:00.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.927+0000 7fe371000640 1 -- 172.21.15.28:0/1961795733 learned_addr learned my addr 172.21.15.28:0/1961795733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe371000640 1 -- 172.21.15.28:0/1961795733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 msgr2=0x7fe36c099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe363e00640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 0x7fe36c099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe371000640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 0x7fe36c099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe36be00640 1 -- 172.21.15.28:0/1961795733 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133932693 0 0) 0x7fe36c0cc1d0 con 0x7fe36c0ccc70 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe371000640 1 -- 172.21.15.28:0/1961795733 >> v1:172.21.15.28:6789/0 conn(0x7fe36c0ccc70 legacy=0x7fe36c0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe371000640 1 -- 172.21.15.28:0/1961795733 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c06d450 con 0x7fe36c15e0c0 2024-03-21T08:43:00.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe363e00640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 0x7fe36c099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:00.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe371000640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c15e0c0 0x7fe36c06cf10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb33a506ca8cc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe36be00640 1 -- 172.21.15.28:0/1961795733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0a95f0 con 0x7fe36c15e0c0 2024-03-21T08:43:00.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe36be00640 1 -- 172.21.15.28:0/1961795733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c0a9790 con 0x7fe36c15e0c0 2024-03-21T08:43:00.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe36be00640 1 -- 172.21.15.28:0/1961795733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0a9b60 con 0x7fe36c15e0c0 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 -- 172.21.15.28:0/1961795733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c15e0c0 msgr2=0x7fe36c06cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c15e0c0 0x7fe36c06cf10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c01c530 comp rx=0 tx=0).stop 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 -- 172.21.15.28:0/1961795733 shutdown_connections 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c099b80 0x7fe36c099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 --2- 172.21.15.28:0/1961795733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c15e0c0 0x7fe36c06cf10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 -- 172.21.15.28:0/1961795733 >> 172.21.15.28:0/1961795733 conn(0x7fe36c0b34d0 msgr2=0x7fe36c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 -- 172.21.15.28:0/1961795733 shutdown_connections 2024-03-21T08:43:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.931+0000 7fe373a00640 1 -- 172.21.15.28:0/1961795733 wait complete. 2024-03-21T08:43:00.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.935+0000 7fe373a00640 1 Processor -- start 2024-03-21T08:43:00.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.935+0000 7fe373a00640 1 -- start start 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c0ccc70 0x7fe36c158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 0x7fe36c15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0d10d0 con 0x7fe36c099b80 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c15e4a0 con 0x7fe36c15e0c0 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c16ac90 con 0x7fe36c0ccc70 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47376/0 (socket says 172.21.15.28:47376) 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 -- 172.21.15.28:0/1390475086 learned_addr learned my addr 172.21.15.28:0/1390475086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 0x7fe36c15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe363e00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c0ccc70 0x7fe36c158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 -- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c0ccc70 msgr2=0x7fe36c158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c0ccc70 0x7fe36c158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 -- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 msgr2=0x7fe36c15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 0x7fe36c15cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 -- 172.21.15.28:0/1390475086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c159640 con 0x7fe36c099b80 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 0x7fe36c15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe371000640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe35c01abe0 tx=0x7fe35c0192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0b18c0 con 0x7fe36c099b80 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c0b1a80 con 0x7fe36c099b80 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0b9430 con 0x7fe36c099b80 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c157bd0 con 0x7fe36c099b80 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c158190 con 0x7fe36c099b80 2024-03-21T08:43:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c04f410 con 0x7fe36c099b80 2024-03-21T08:43:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.939+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe35c0b95d0 con 0x7fe36c099b80 2024-03-21T08:43:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.943+0000 7fe369600640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 0x7fe34806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.943+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe35c120820 con 0x7fe36c099b80 2024-03-21T08:43:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.943+0000 7fe363e00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 0x7fe34806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.943+0000 7fe363e00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 0x7fe34806a030 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe354006fd0 tx=0x7fe354008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:00.947+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe35c0ad020 con 0x7fe36c099b80 2024-03-21T08:43:01.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.111+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe36c0cd440 con 0x7fe36c099b80 2024-03-21T08:43:01.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.111+0000 7fe369600640 1 -- 172.21.15.28:0/1390475086 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7fe35c016150 con 0x7fe36c099b80 2024-03-21T08:43:01.114 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:01.114 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 msgr2=0x7fe34806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 0x7fe34806a030 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe354006fd0 tx=0x7fe354008040 comp rx=0 tx=0).stop 2024-03-21T08:43:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 msgr2=0x7fe36c15c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe35c01abe0 tx=0x7fe35c0192d0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 shutdown_connections 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe36c15e0c0 0x7fe36c15cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe348067b70 0x7fe34806a030 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe36c0ccc70 0x7fe36c158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 --2- 172.21.15.28:0/1390475086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c099b80 0x7fe36c15c400 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 >> 172.21.15.28:0/1390475086 conn(0x7fe36c0b34d0 msgr2=0x7fe36c06fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 shutdown_connections 2024-03-21T08:43:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.115+0000 7fe373a00640 1 -- 172.21.15.28:0/1390475086 wait complete. 2024-03-21T08:43:01.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:01.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:01.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 Processor -- start 2024-03-21T08:43:01.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 -- start start 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc0b0e80 0x7f27fc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f27fc0cea00 0x7f27fc08afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc15fba0 con 0x7f27fc15bd20 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0cee50 con 0x7f27fc0cea00 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2804000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc15fe80 con 0x7f27fc0b0e80 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2801600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f27fc15bd20 0x7f27fc0b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46608/0 (socket says 172.21.15.28:46608) 2024-03-21T08:43:01.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2801600640 1 -- 172.21.15.28:0/1147787165 learned_addr learned my addr 172.21.15.28:0/1147787165 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2802000640 1 --2- 172.21.15.28:0/1147787165 >> v2:172.21.15.151:3300/0 conn(0x7f27fc0cea00 0x7f27fc08afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2800c00640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc0b0e80 0x7f27fc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240258256 0 0) 0x7f27fc15fba0 con 0x7f27fc15bd20 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003530 con 0x7f27fc15bd20 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2802000640 1 -- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc0b0e80 msgr2=0x7f27fc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2802000640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc0b0e80 0x7f27fc0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2802000640 1 -- 172.21.15.28:0/1147787165 >> v1:172.21.15.28:6789/0 conn(0x7f27fc15bd20 legacy=0x7f27fc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2905083925 0 0) 0x7f27e4003530 con 0x7f27fc15bd20 2024-03-21T08:43:01.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.279+0000 7f2802000640 1 -- 172.21.15.28:0/1147787165 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc160160 con 0x7f27fc0cea00 2024-03-21T08:43:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2802000640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc08afc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009d90 tx=0x7f27f401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70d47d0dd96c0c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f40b2070 con 0x7f27fc0cea00 2024-03-21T08:43:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f40a55a0 con 0x7f27fc0cea00 2024-03-21T08:43:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f27fbe00640 1 -- 172.21.15.28:0/1147787165 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0001680 con 0x7f27fc0cea00 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 -- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 msgr2=0x7f27fc08afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc08afc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009d90 tx=0x7f27f401c890 comp rx=0 tx=0).stop 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 -- 172.21.15.28:0/1147787165 shutdown_connections 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc08afc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 --2- 172.21.15.28:0/1147787165 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc0b0e80 0x7f27fc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 -- 172.21.15.28:0/1147787165 >> 172.21.15.28:0/1147787165 conn(0x7f27fc0d1ab0 msgr2=0x7f27fc0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 -- 172.21.15.28:0/1147787165 shutdown_connections 2024-03-21T08:43:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.283+0000 7f2804000640 1 -- 172.21.15.28:0/1147787165 wait complete. 2024-03-21T08:43:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 Processor -- start 2024-03-21T08:43:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- start start 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc15bd20 0x7f27fc1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc08be90 con 0x7f27fc0b0e80 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc08bd10 con 0x7f27fc0cea00 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc168d10 con 0x7f27fc15bd20 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47378/0 (socket says 172.21.15.28:47378) 2024-03-21T08:43:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 -- 172.21.15.28:0/187940966 learned_addr learned my addr 172.21.15.28:0/187940966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2800c00640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2802000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc15bd20 0x7f27fc1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 -- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc15bd20 msgr2=0x7f27fc1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc15bd20 0x7f27fc1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 -- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 msgr2=0x7f27fc158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 -- 172.21.15.28:0/187940966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0c95f0 con 0x7f27fc0b0e80 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2801600640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f27f00017f0 tx=0x7f27f000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0005ae0 con 0x7f27fc0b0e80 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0005ca0 con 0x7f27fc0b0e80 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc0c9890 con 0x7f27fc0b0e80 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00072b0 con 0x7f27fc0b0e80 2024-03-21T08:43:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.287+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc0c9e50 con 0x7f27fc0b0e80 2024-03-21T08:43:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.291+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0ca340 con 0x7f27fc0b0e80 2024-03-21T08:43:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.291+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f27f0007450 con 0x7f27fc0b0e80 2024-03-21T08:43:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.291+0000 7f27f9600640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 0x7f27dc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.291+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f27f0085e50 con 0x7f27fc0b0e80 2024-03-21T08:43:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.291+0000 7f2800c00640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 0x7f27dc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.295+0000 7f2800c00640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 0x7f27dc069fe0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ce0d0 tx=0x7f27e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.295+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f27f0055f10 con 0x7f27fc0b0e80 2024-03-21T08:43:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.459+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27fc071a50 con 0x7f27fc0b0e80 2024-03-21T08:43:01.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.459+0000 7f27f9600640 1 -- 172.21.15.28:0/187940966 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27f0059fb0 con 0x7f27fc0b0e80 2024-03-21T08:43:01.464 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 msgr2=0x7f27dc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 0x7f27dc069fe0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ce0d0 tx=0x7f27e8009190 comp rx=0 tx=0).stop 2024-03-21T08:43:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 msgr2=0x7f27fc155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f27f00017f0 tx=0x7f27f000cbe0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 shutdown_connections 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27dc067b20 0x7f27dc069fe0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f27fc15bd20 0x7f27fc1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f27fc0cea00 0x7f27fc158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 --2- 172.21.15.28:0/187940966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc0b0e80 0x7f27fc155c50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 >> 172.21.15.28:0/187940966 conn(0x7f27fc0d1ab0 msgr2=0x7f27fc1658a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 shutdown_connections 2024-03-21T08:43:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.463+0000 7f2804000640 1 -- 172.21.15.28:0/187940966 wait complete. 2024-03-21T08:43:01.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:01.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 Processor -- start 2024-03-21T08:43:01.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 -- start start 2024-03-21T08:43:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256015dff0 0x7f2560099c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25600b1110 0x7f256009a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 0x7f256006d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25600cc180 con 0x7f25600b1110 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25600cc3d0 con 0x7f256006cc00 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f2565000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25600cc6b0 con 0x7f256015dff0 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25600b1110 0x7f256009a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25600b1110 0x7f256009a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47394/0 (socket says 172.21.15.28:47394) 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 -- 172.21.15.28:0/2023731349 learned_addr learned my addr 172.21.15.28:0/2023731349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255de00640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256015dff0 0x7f2560099c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255e800640 1 --2- 172.21.15.28:0/2023731349 >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 0x7f256006d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 -- 172.21.15.28:0/2023731349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256015dff0 msgr2=0x7f2560099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256015dff0 0x7f2560099c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 -- 172.21.15.28:0/2023731349 >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 msgr2=0x7f256006d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 --2- 172.21.15.28:0/2023731349 >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 0x7f256006d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255d400640 1 -- 172.21.15.28:0/2023731349 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25600cc990 con 0x7f25600b1110 2024-03-21T08:43:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.627+0000 7f255e800640 1 --2- 172.21.15.28:0/2023731349 >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 0x7f256006d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f255d400640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f256009a1d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f25600d2b80 tx=0x7f254c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588f70db00399bb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f255ca00640 1 -- 172.21.15.28:0/2023731349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0b0070 con 0x7f25600b1110 2024-03-21T08:43:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f255ca00640 1 -- 172.21.15.28:0/2023731349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c0ab600 con 0x7f25600b1110 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f255ca00640 1 -- 172.21.15.28:0/2023731349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0ab9d0 con 0x7f25600b1110 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- 172.21.15.28:0/2023731349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 msgr2=0x7f256009a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f256009a1d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f25600d2b80 tx=0x7f254c01c400 comp rx=0 tx=0).stop 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- 172.21.15.28:0/2023731349 shutdown_connections 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 --2- 172.21.15.28:0/2023731349 >> v2:172.21.15.151:3300/0 conn(0x7f256006cc00 0x7f256006d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f256009a1d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 --2- 172.21.15.28:0/2023731349 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256015dff0 0x7f2560099c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- 172.21.15.28:0/2023731349 >> 172.21.15.28:0/2023731349 conn(0x7f25600b3620 msgr2=0x7f25600b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- 172.21.15.28:0/2023731349 shutdown_connections 2024-03-21T08:43:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- 172.21.15.28:0/2023731349 wait complete. 2024-03-21T08:43:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 Processor -- start 2024-03-21T08:43:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.631+0000 7f2565000640 1 -- start start 2024-03-21T08:43:01.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256006cc00 0x7f2560155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 0x7f25600ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600cdd30 con 0x7f25600b1110 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600cdbb0 con 0x7f256015dff0 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25600cdeb0 con 0x7f256006cc00 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47404/0 (socket says 172.21.15.28:47404) 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 -- 172.21.15.28:0/2599943152 learned_addr learned my addr 172.21.15.28:0/2599943152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255de00640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256006cc00 0x7f2560155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255e800640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 0x7f25600ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 -- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256006cc00 msgr2=0x7f2560155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256006cc00 0x7f2560155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 -- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 msgr2=0x7f25600ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 0x7f25600ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 -- 172.21.15.28:0/2599943152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25600c95a0 con 0x7f25600b1110 2024-03-21T08:43:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255e800640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 0x7f25600ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f255d400640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f254c01aab0 tx=0x7f254c0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0b0040 con 0x7f25600b1110 2024-03-21T08:43:01.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c0b9070 con 0x7f25600b1110 2024-03-21T08:43:01.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0b4660 con 0x7f25600b1110 2024-03-21T08:43:01.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25600c9880 con 0x7f25600b1110 2024-03-21T08:43:01.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25600c9e40 con 0x7f25600b1110 2024-03-21T08:43:01.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.635+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256009faf0 con 0x7f25600b1110 2024-03-21T08:43:01.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.639+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f254c0b4800 con 0x7f25600b1110 2024-03-21T08:43:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.639+0000 7f2556000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 0x7f2540069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.639+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f254c11fc70 con 0x7f25600b1110 2024-03-21T08:43:01.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.639+0000 7f255de00640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 0x7f2540069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.639+0000 7f255de00640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 0x7f2540069fe0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2550009b60 tx=0x7f2550009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.643+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f254c016110 con 0x7f25600b1110 2024-03-21T08:43:01.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.807+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2560074270 con 0x7f25600b1110 2024-03-21T08:43:01.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.807+0000 7f2556000640 1 -- 172.21.15.28:0/2599943152 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f254c0f04a0 con 0x7f25600b1110 2024-03-21T08:43:01.811 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 msgr2=0x7f2540069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 0x7f2540069fe0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2550009b60 tx=0x7f2550009190 comp rx=0 tx=0).stop 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 msgr2=0x7f25600c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f254c01aab0 tx=0x7f254c0191a0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 shutdown_connections 2024-03-21T08:43:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f256015dff0 0x7f25600ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25600b1110 0x7f25600c8c00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2540067b20 0x7f2540069fe0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 --2- 172.21.15.28:0/2599943152 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f256006cc00 0x7f2560155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 >> 172.21.15.28:0/2599943152 conn(0x7f25600b3620 msgr2=0x7f25600d0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 shutdown_connections 2024-03-21T08:43:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.811+0000 7f2565000640 1 -- 172.21.15.28:0/2599943152 wait complete. 2024-03-21T08:43:01.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.979+0000 7ffbb9600640 1 Processor -- start 2024-03-21T08:43:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.979+0000 7ffbb9600640 1 -- start start 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb9600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ffbb415bd20 0x7ffbb40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbb40cebd0 0x7ffbb408afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb9600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb415fba0 con 0x7ffbb40b0e80 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb415fe80 con 0x7ffbb40cebd0 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb9600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb4160160 con 0x7ffbb415bd20 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbb40cebd0 0x7ffbb408afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb2000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbb40b0e80 0x7ffbb40ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46616/0 (socket says 172.21.15.28:46616) 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb2000640 1 -- 172.21.15.28:0/511769901 learned_addr learned my addr 172.21.15.28:0/511769901 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbb40cebd0 0x7ffbb408afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47416/0 (socket says 172.21.15.28:47416) 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb2a00640 1 --2- 172.21.15.28:0/511769901 >> v2:172.21.15.151:3300/0 conn(0x7ffbb415bd20 0x7ffbb40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 -- 172.21.15.28:0/511769901 >> v2:172.21.15.151:3300/0 conn(0x7ffbb415bd20 msgr2=0x7ffbb40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 --2- 172.21.15.28:0/511769901 >> v2:172.21.15.151:3300/0 conn(0x7ffbb415bd20 0x7ffbb40b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 -- 172.21.15.28:0/511769901 >> v1:172.21.15.28:6789/0 conn(0x7ffbb40b0e80 legacy=0x7ffbb40ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb3400640 1 -- 172.21.15.28:0/511769901 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb4160440 con 0x7ffbb40cebd0 2024-03-21T08:43:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.983+0000 7ffbb1600640 1 -- 172.21.15.28:0/511769901 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253815347 0 0) 0x7ffbb415fba0 con 0x7ffbb40b0e80 2024-03-21T08:43:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb3400640 1 --2- 172.21.15.28:0/511769901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb408afc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4068b20 tx=0x7ffbac01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0bdfe77b025671d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb1600640 1 -- 172.21.15.28:0/511769901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0b2070 con 0x7ffbb40cebd0 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb1600640 1 -- 172.21.15.28:0/511769901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac0a57f0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb1600640 1 -- 172.21.15.28:0/511769901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0a5bc0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- 172.21.15.28:0/511769901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 msgr2=0x7ffbb408afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- 172.21.15.28:0/511769901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb408afc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4068b20 tx=0x7ffbac01c930 comp rx=0 tx=0).stop 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- 172.21.15.28:0/511769901 shutdown_connections 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- 172.21.15.28:0/511769901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb408afc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- 172.21.15.28:0/511769901 >> v2:172.21.15.151:3300/0 conn(0x7ffbb415bd20 0x7ffbb40b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- 172.21.15.28:0/511769901 >> 172.21.15.28:0/511769901 conn(0x7ffbb40d1ab0 msgr2=0x7ffbb40d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- 172.21.15.28:0/511769901 shutdown_connections 2024-03-21T08:43:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- 172.21.15.28:0/511769901 wait complete. 2024-03-21T08:43:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 Processor -- start 2024-03-21T08:43:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- start start 2024-03-21T08:43:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 0x7ffbb40c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb415bd20 0x7ffbb40c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb408bd10 con 0x7ffbb40cebd0 2024-03-21T08:43:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb415c310 con 0x7ffbb415bd20 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.987+0000 7ffbb9600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb408be90 con 0x7ffbb40b0e80 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47426/0 (socket says 172.21.15.28:47426) 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 -- 172.21.15.28:0/1338240340 learned_addr learned my addr 172.21.15.28:0/1338240340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb3400640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb415bd20 0x7ffbb40c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2a00640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 0x7ffbb40c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 -- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 msgr2=0x7ffbb40c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 0x7ffbb40c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 -- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb415bd20 msgr2=0x7ffbb40c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb415bd20 0x7ffbb40c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 -- 172.21.15.28:0/1338240340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb40ca2d0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2a00640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 0x7ffbb40c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb2000640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ffba000c480 tx=0x7ffba000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0018070 con 0x7ffbb40cebd0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0007500 con 0x7ffbb40cebd0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0004a30 con 0x7ffbb40cebd0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb40ca5b0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb40cab70 con 0x7ffbb40cebd0 2024-03-21T08:43:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.991+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb404f4f0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.995+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ffba0004bd0 con 0x7ffbb40cebd0 2024-03-21T08:43:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.995+0000 7ffbaaa00640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 0x7ffb9406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.995+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffba0085460 con 0x7ffbb40cebd0 2024-03-21T08:43:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.995+0000 7ffbb2a00640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 0x7ffb9406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.995+0000 7ffbb2a00640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 0x7ffb9406a010 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009840 tx=0x7ffba4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:01.999+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffba0055b70 con 0x7ffbb40cebd0 2024-03-21T08:43:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.159+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbb40d2830 con 0x7ffbb40cebd0 2024-03-21T08:43:02.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.159+0000 7ffbaaa00640 1 -- 172.21.15.28:0/1338240340 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffba0055470 con 0x7ffbb40cebd0 2024-03-21T08:43:02.163 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 msgr2=0x7ffb9406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 0x7ffb9406a010 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009840 tx=0x7ffba4008040 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 msgr2=0x7ffbb406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ffba000c480 tx=0x7ffba000c940 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 shutdown_connections 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ffbb415bd20 0x7ffbb40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb40cebd0 0x7ffbb406ddb0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ffb94067b50 0x7ffb9406a010 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 --2- 172.21.15.28:0/1338240340 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ffbb40b0e80 0x7ffbb40c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 >> 172.21.15.28:0/1338240340 conn(0x7ffbb40d1ab0 msgr2=0x7ffbb40d1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 shutdown_connections 2024-03-21T08:43:02.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.163+0000 7ffbb9600640 1 -- 172.21.15.28:0/1338240340 wait complete. 2024-03-21T08:43:02.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:43:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 Processor -- start 2024-03-21T08:43:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 -- start start 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77840ce590 0x7f77840d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77840d0ad0 0x7f77840cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840d0f60 con 0x7f778415aa30 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840cf930 con 0x7f77840ce590 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f778ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778415be50 con 0x7f77840d0ad0 2024-03-21T08:43:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f777b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77840ce590 0x7f77840d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f777b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77840ce590 0x7f77840d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47428/0 (socket says 172.21.15.28:47428) 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.327+0000 7f777b400640 1 -- 172.21.15.28:0/3652583699 learned_addr learned my addr 172.21.15.28:0/3652583699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f7788e00640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77840d0ad0 0x7f77840cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f777b400640 1 -- 172.21.15.28:0/3652583699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77840d0ad0 msgr2=0x7f77840cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f777b400640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77840d0ad0 0x7f77840cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f777b400640 1 -- 172.21.15.28:0/3652583699 >> v1:172.21.15.151:6789/0 conn(0x7f778415aa30 legacy=0x7f778415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f777b400640 1 -- 172.21.15.28:0/3652583699 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778415b890 con 0x7f77840ce590 2024-03-21T08:43:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f777b400640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840ce590 0x7f77840d0590 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f776c009e80 tx=0x7f776c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cd11596b2b4773 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f7783400640 1 -- 172.21.15.28:0/3652583699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0a5810 con 0x7f77840ce590 2024-03-21T08:43:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f7783400640 1 -- 172.21.15.28:0/3652583699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c0a59b0 con 0x7f77840ce590 2024-03-21T08:43:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.331+0000 7f7783400640 1 -- 172.21.15.28:0/3652583699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0a5d80 con 0x7f77840ce590 2024-03-21T08:43:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- 172.21.15.28:0/3652583699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840ce590 msgr2=0x7f77840d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840ce590 0x7f77840d0590 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f776c009e80 tx=0x7f776c01c950 comp rx=0 tx=0).stop 2024-03-21T08:43:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- 172.21.15.28:0/3652583699 shutdown_connections 2024-03-21T08:43:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f77840d0ad0 0x7f77840cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 --2- 172.21.15.28:0/3652583699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840ce590 0x7f77840d0590 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- 172.21.15.28:0/3652583699 >> 172.21.15.28:0/3652583699 conn(0x7f7784099a30 msgr2=0x7f7784099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- 172.21.15.28:0/3652583699 shutdown_connections 2024-03-21T08:43:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- 172.21.15.28:0/3652583699 wait complete. 2024-03-21T08:43:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 Processor -- start 2024-03-21T08:43:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.335+0000 7f778ae00640 1 -- start start 2024-03-21T08:43:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 0x7f77840af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f778415aa30 0x7f77840cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840ce070 con 0x7f77840d0ad0 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778415b520 con 0x7f77840ce590 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840cbc00 con 0x7f778415aa30 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47440/0 (socket says 172.21.15.28:47440) 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 -- 172.21.15.28:0/892414153 learned_addr learned my addr 172.21.15.28:0/892414153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7783e00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 0x7f77840af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7788e00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f778415aa30 0x7f77840cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 -- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f778415aa30 msgr2=0x7f77840cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f778415aa30 0x7f77840cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 -- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 msgr2=0x7f77840af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 0x7f77840af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 -- 172.21.15.28:0/892414153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840ad6e0 con 0x7f77840d0ad0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7783e00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 0x7f77840af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f777b400640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f776c01a580 tx=0x7f776c0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0bc020 con 0x7f77840d0ad0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c0b8670 con 0x7f77840d0ad0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7784152900 con 0x7f77840d0ad0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0b8a20 con 0x7f77840d0ad0 2024-03-21T08:43:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77841f7840 con 0x7f77840d0ad0 2024-03-21T08:43:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.339+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778409ea60 con 0x7f77840d0ad0 2024-03-21T08:43:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.343+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f776c0bf020 con 0x7f77840d0ad0 2024-03-21T08:43:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.343+0000 7f7780c00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 0x7f7760069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.343+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f776c1213c0 con 0x7f77840d0ad0 2024-03-21T08:43:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.343+0000 7f7783e00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 0x7f7760069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.343+0000 7f7783e00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 0x7f7760069fe0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7774000f70 tx=0x7f7774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.347+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f776c0c6030 con 0x7f77840d0ad0 2024-03-21T08:43:02.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.543+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f778404f4f0 con 0x7f77840d0ad0 2024-03-21T08:43:02.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.543+0000 7f7780c00640 1 -- 172.21.15.28:0/892414153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7f7784152900 con 0x7f77840d0ad0 2024-03-21T08:43:02.547 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 msgr2=0x7f7760069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 0x7f7760069fe0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7774000f70 tx=0x7f7774009040 comp rx=0 tx=0).stop 2024-03-21T08:43:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 msgr2=0x7f77841f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f776c01a580 tx=0x7f776c0196f0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 shutdown_connections 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7760067b20 0x7f7760069fe0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f778415aa30 0x7f77840cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77840d0ad0 0x7f77841f6e40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 --2- 172.21.15.28:0/892414153 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f77840ce590 0x7f77840af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 >> 172.21.15.28:0/892414153 conn(0x7f7784099a30 msgr2=0x7f778415ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 shutdown_connections 2024-03-21T08:43:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.547+0000 7f778ae00640 1 -- 172.21.15.28:0/892414153 wait complete. 2024-03-21T08:43:02.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.711+0000 7f6ce6c00640 1 Processor -- start 2024-03-21T08:43:02.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 -- start start 2024-03-21T08:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ce015bdc0 0x7f6ce00cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce0099c90 0x7f6ce009a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce00ccbd0 con 0x7f6ce00cc640 2024-03-21T08:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce00cd7e0 con 0x7f6ce015bdc0 2024-03-21T08:43:02.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce6c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00cceb0 con 0x7f6ce0099c90 2024-03-21T08:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cd7400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f6ce00cc640 0x7f6ce00d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56274/0 (socket says 172.21.15.28:56274) 2024-03-21T08:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdfe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ce015bdc0 0x7f6ce00cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6ce4c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce0099c90 0x7f6ce009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cd7400640 1 -- 172.21.15.28:0/1482867199 learned_addr learned my addr 172.21.15.28:0/1482867199 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:02.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdf400640 1 -- 172.21.15.28:0/1482867199 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228404742 0 0) 0x7f6ce00ccbd0 con 0x7f6ce00cc640 2024-03-21T08:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdf400640 1 -- 172.21.15.28:0/1482867199 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4003530 con 0x7f6ce00cc640 2024-03-21T08:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdfe00640 1 -- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce0099c90 msgr2=0x7f6ce009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdfe00640 1 --2- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce0099c90 0x7f6ce009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdfe00640 1 -- 172.21.15.28:0/1482867199 >> v1:172.21.15.151:6789/0 conn(0x7f6ce00cc640 legacy=0x7f6ce00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.715+0000 7f6cdfe00640 1 -- 172.21.15.28:0/1482867199 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00cd190 con 0x7f6ce015bdc0 2024-03-21T08:43:02.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6cdfe00640 1 --2- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce015bdc0 0x7f6ce00cdc70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00097c0 tx=0x7f6cd001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb39fbd3aaa3c90e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6cdf400640 1 -- 172.21.15.28:0/1482867199 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00b0070 con 0x7f6ce015bdc0 2024-03-21T08:43:02.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6cdf400640 1 -- 172.21.15.28:0/1482867199 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd00ab600 con 0x7f6ce015bdc0 2024-03-21T08:43:02.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6cdf400640 1 -- 172.21.15.28:0/1482867199 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00ab9b0 con 0x7f6ce015bdc0 2024-03-21T08:43:02.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce015bdc0 msgr2=0x7f6ce00cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce015bdc0 0x7f6ce00cdc70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00097c0 tx=0x7f6cd001c410 comp rx=0 tx=0).stop 2024-03-21T08:43:02.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- 172.21.15.28:0/1482867199 shutdown_connections 2024-03-21T08:43:02.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce0099c90 0x7f6ce009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/1482867199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce015bdc0 0x7f6ce00cdc70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- 172.21.15.28:0/1482867199 >> 172.21.15.28:0/1482867199 conn(0x7f6ce00b3620 msgr2=0x7f6ce00b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- 172.21.15.28:0/1482867199 shutdown_connections 2024-03-21T08:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- 172.21.15.28:0/1482867199 wait complete. 2024-03-21T08:43:02.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 Processor -- start 2024-03-21T08:43:02.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.719+0000 7f6ce6c00640 1 -- start start 2024-03-21T08:43:02.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 0x7f6ce0158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 0x7f6ce0159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00b1cc0 con 0x7f6ce0099c90 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00b1b40 con 0x7f6ce00cc640 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00b1e40 con 0x7f6ce015bdc0 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47448/0 (socket says 172.21.15.28:47448) 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 -- 172.21.15.28:0/4283836696 learned_addr learned my addr 172.21.15.28:0/4283836696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cd7400640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 0x7f6ce0158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce4c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 0x7f6ce0159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 -- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 msgr2=0x7f6ce0159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 0x7f6ce0159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 -- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 msgr2=0x7f6ce0158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 0x7f6ce0158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 -- 172.21.15.28:0/4283836696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0159da0 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cd7400640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 0x7f6ce0158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce4c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 0x7f6ce0159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdfe00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00ae040 tx=0x7f6cd001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00b0050 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd00bc070 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0006e20 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce015a080 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce015a640 con 0x7f6ce0099c90 2024-03-21T08:43:02.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.723+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce009faf0 con 0x7f6ce0099c90 2024-03-21T08:43:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.727+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6cd00b5830 con 0x7f6ce0099c90 2024-03-21T08:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.727+0000 7f6cdcc00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 0x7f6cbc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.727+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6cd01208d0 con 0x7f6ce0099c90 2024-03-21T08:43:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.727+0000 7f6cd7400640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 0x7f6cbc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:02.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.727+0000 7f6cd7400640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 0x7f6cbc069fe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0151f30 tx=0x7f6cc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:02.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.731+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6cd0016090 con 0x7f6ce0099c90 2024-03-21T08:43:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.907+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6ce004f4f0 con 0x7f6ce0099c90 2024-03-21T08:43:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.907+0000 7f6cdcc00640 1 -- 172.21.15.28:0/4283836696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6cd00f10f0 con 0x7f6ce0099c90 2024-03-21T08:43:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 msgr2=0x7f6cbc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 0x7f6cbc069fe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0151f30 tx=0x7f6cc8009040 comp rx=0 tx=0).stop 2024-03-21T08:43:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 msgr2=0x7f6ce0070fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00ae040 tx=0x7f6cd001ac50 comp rx=0 tx=0).stop 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 shutdown_connections 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6cbc067b20 0x7f6cbc069fe0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ce015bdc0 0x7f6ce0159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ce00cc640 0x7f6ce0158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 --2- 172.21.15.28:0/4283836696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce0099c90 0x7f6ce0070fa0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 >> 172.21.15.28:0/4283836696 conn(0x7f6ce00b3620 msgr2=0x7f6ce00af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 shutdown_connections 2024-03-21T08:43:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:02.911+0000 7f6ce6c00640 1 -- 172.21.15.28:0/4283836696 wait complete. 2024-03-21T08:43:02.937 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:43:02.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:02.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:02.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38094]: fuse finished with error 0 and tester_r 0 2024-03-21T08:43:02.958 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:02.981 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:43:02.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:02.981 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:43:03.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-21T08:43:03.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 Processor -- start 2024-03-21T08:43:03.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 -- start start 2024-03-21T08:43:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe49409a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe4940cc540 0x7fe4940cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49409b1e0 con 0x7fe49415e2a0 2024-03-21T08:43:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49409a630 con 0x7fe4940cc540 2024-03-21T08:43:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe49b600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4940d0930 con 0x7fe494099c90 2024-03-21T08:43:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe499600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe4940cc540 0x7fe4940cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe498c00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fe49415e2a0 0x7fe49409aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56280/0 (socket says 172.21.15.28:56280) 2024-03-21T08:43:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe48be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe49409a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.219+0000 7fe498c00640 1 -- 172.21.15.28:0/3780563095 learned_addr learned my addr 172.21.15.28:0/3780563095 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe493e00640 1 -- 172.21.15.28:0/3780563095 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263578083 0 0) 0x7fe484001120 con 0x7fe49415e2a0 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe493e00640 1 -- 172.21.15.28:0/3780563095 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe478003530 con 0x7fe49415e2a0 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe499600640 1 -- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 msgr2=0x7fe49409a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe499600640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe49409a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe499600640 1 -- 172.21.15.28:0/3780563095 >> v1:172.21.15.151:6789/0 conn(0x7fe49415e2a0 legacy=0x7fe49409aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe499600640 1 -- 172.21.15.28:0/3780563095 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4940d0c10 con 0x7fe4940cc540 2024-03-21T08:43:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe48be00640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe49409a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe499600640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe4940cc9a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009d90 tx=0x7fe48c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f12f9e0b18592e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe493e00640 1 -- 172.21.15.28:0/3780563095 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0b2070 con 0x7fe4940cc540 2024-03-21T08:43:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe493e00640 1 -- 172.21.15.28:0/3780563095 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c0a5890 con 0x7fe4940cc540 2024-03-21T08:43:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.223+0000 7fe493e00640 1 -- 172.21.15.28:0/3780563095 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0a5c60 con 0x7fe4940cc540 2024-03-21T08:43:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 msgr2=0x7fe4940cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe4940cc9a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009d90 tx=0x7fe48c01c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- 172.21.15.28:0/3780563095 shutdown_connections 2024-03-21T08:43:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe4940cc9a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 --2- 172.21.15.28:0/3780563095 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe49409a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- 172.21.15.28:0/3780563095 >> 172.21.15.28:0/3780563095 conn(0x7fe4940b3620 msgr2=0x7fe4940b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- 172.21.15.28:0/3780563095 shutdown_connections 2024-03-21T08:43:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- 172.21.15.28:0/3780563095 wait complete. 2024-03-21T08:43:03.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 Processor -- start 2024-03-21T08:43:03.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.227+0000 7fe49b600640 1 -- start start 2024-03-21T08:43:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe494151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe494158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49415e680 con 0x7fe49415e2a0 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49416aee0 con 0x7fe4940cc540 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe494152640 con 0x7fe494099c90 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe498c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe494151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47462/0 (socket says 172.21.15.28:47462) 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 -- 172.21.15.28:0/2500676992 learned_addr learned my addr 172.21.15.28:0/2500676992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe48be00640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe494158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 -- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 msgr2=0x7fe494151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe494151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 -- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 msgr2=0x7fe494158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe494158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 -- 172.21.15.28:0/2500676992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494152920 con 0x7fe49415e2a0 2024-03-21T08:43:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe499600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe48c01af90 tx=0x7fe48c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0b2050 con 0x7fe49415e2a0 2024-03-21T08:43:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c0be070 con 0x7fe49415e2a0 2024-03-21T08:43:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe494152c00 con 0x7fe49415e2a0 2024-03-21T08:43:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0ad870 con 0x7fe49415e2a0 2024-03-21T08:43:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4941531c0 con 0x7fe49415e2a0 2024-03-21T08:43:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.231+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49409faf0 con 0x7fe49415e2a0 2024-03-21T08:43:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.235+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe48c0ada10 con 0x7fe49415e2a0 2024-03-21T08:43:03.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.235+0000 7fe491600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 0x7fe470069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.235+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe48c1212f0 con 0x7fe49415e2a0 2024-03-21T08:43:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.235+0000 7fe498c00640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 0x7fe470069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.235+0000 7fe498c00640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 0x7fe470069fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe484005890 tx=0x7fe484009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.239+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe48c016560 con 0x7fe49415e2a0 2024-03-21T08:43:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.463+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7fe4940c8970 con 0x7fe49415e2a0 2024-03-21T08:43:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe491600640 1 -- 172.21.15.28:0/2500676992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v13) v1 ==== 211+0+168 (secure 0 0 0) 0x7fe48c0add00 con 0x7fe49415e2a0 2024-03-21T08:43:03.467 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:43:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:43:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow rwp" 2024-03-21T08:43:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:43:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-21T08:43:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:43:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 msgr2=0x7fe470069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 0x7fe470069fe0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe484005890 tx=0x7fe484009040 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 msgr2=0x7fe4941591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe48c01af90 tx=0x7fe48c019680 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 shutdown_connections 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49415e2a0 0x7fe4941591c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe4940cc540 0x7fe494158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe470067b20 0x7fe470069fe0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 --2- 172.21.15.28:0/2500676992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe494099c90 0x7fe494151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 >> 172.21.15.28:0/2500676992 conn(0x7fe4940b3620 msgr2=0x7fe49415c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 shutdown_connections 2024-03-21T08:43:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.467+0000 7fe49b600640 1 -- 172.21.15.28:0/2500676992 wait complete. 2024-03-21T08:43:03.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:43:03.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.627+0000 7fd9ad200640 1 Processor -- start 2024-03-21T08:43:03.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9ad200640 1 -- start start 2024-03-21T08:43:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9ad200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 0x7fd9a809af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9ad200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a80d0780 con 0x7fd9a809a290 2024-03-21T08:43:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9ad200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a80cc810 con 0x7fd9a80b1360 2024-03-21T08:43:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9ad200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80ccaf0 con 0x7fd9a815d060 2024-03-21T08:43:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9a809a290 0x7fd9a80cd2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46626/0 (socket says 172.21.15.28:46626) 2024-03-21T08:43:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a6800640 1 -- 172.21.15.28:0/2216514097 learned_addr learned my addr 172.21.15.28:0/2216514097 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5e00640 1 --2- 172.21.15.28:0/2216514097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 0x7fd9a809af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859510483 0 0) 0x7fd9a80d0780 con 0x7fd9a809a290 2024-03-21T08:43:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd98c003530 con 0x7fd9a809a290 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824729082 0 0) 0x7fd9a80cc810 con 0x7fd9a80b1360 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a80d0780 con 0x7fd9a80b1360 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5e00640 1 -- 172.21.15.28:0/2216514097 >> v1:172.21.15.151:6789/0 conn(0x7fd9a80b1360 legacy=0x7fd9a8099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5e00640 1 -- 172.21.15.28:0/2216514097 >> v1:172.21.15.28:6789/0 conn(0x7fd9a809a290 legacy=0x7fd9a80cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777544552 0 0) 0x7fd98c003530 con 0x7fd9a809a290 2024-03-21T08:43:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.631+0000 7fd9a5e00640 1 -- 172.21.15.28:0/2216514097 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80ccdd0 con 0x7fd9a815d060 2024-03-21T08:43:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9a5e00640 1 --2- 172.21.15.28:0/2216514097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 0x7fd9a809af80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd998009df0 tx=0x7fd99801c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1666962af8ba7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9980b2070 con 0x7fd9a815d060 2024-03-21T08:43:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9980a5a70 con 0x7fd9a815d060 2024-03-21T08:43:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9a5400640 1 -- 172.21.15.28:0/2216514097 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9980a5e40 con 0x7fd9a815d060 2024-03-21T08:43:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- 172.21.15.28:0/2216514097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 msgr2=0x7fd9a809af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 --2- 172.21.15.28:0/2216514097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 0x7fd9a809af80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd998009df0 tx=0x7fd99801c950 comp rx=0 tx=0).stop 2024-03-21T08:43:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- 172.21.15.28:0/2216514097 shutdown_connections 2024-03-21T08:43:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 --2- 172.21.15.28:0/2216514097 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a815d060 0x7fd9a809af80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- 172.21.15.28:0/2216514097 >> 172.21.15.28:0/2216514097 conn(0x7fd9a80b3510 msgr2=0x7fd9a80b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- 172.21.15.28:0/2216514097 shutdown_connections 2024-03-21T08:43:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- 172.21.15.28:0/2216514097 wait complete. 2024-03-21T08:43:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 Processor -- start 2024-03-21T08:43:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.635+0000 7fd9ad200640 1 -- start start 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a809a290 0x7fd9a806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9a815d060 0x7fd9a81598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80b0c00 con 0x7fd9a80b1360 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80b1b50 con 0x7fd9a815d060 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a80d0330 con 0x7fd9a809a290 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47476/0 (socket says 172.21.15.28:47476) 2024-03-21T08:43:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 -- 172.21.15.28:0/168114482 learned_addr learned my addr 172.21.15.28:0/168114482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9a5e00640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a809a290 0x7fd9a806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9a6800640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9a815d060 0x7fd9a81598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 -- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a809a290 msgr2=0x7fd9a806d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a809a290 0x7fd9a806d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 -- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9a815d060 msgr2=0x7fd9a81598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9a815d060 0x7fd9a81598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 -- 172.21.15.28:0/168114482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a806ce30 con 0x7fd9a80b1360 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99d400640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c490 tx=0x7fd99000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9900052d0 con 0x7fd9a80b1360 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd990005470 con 0x7fd9a80b1360 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a80b5930 con 0x7fd9a80b1360 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990012870 con 0x7fd9a80b1360 2024-03-21T08:43:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a8159e20 con 0x7fd9a80b1360 2024-03-21T08:43:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.639+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a804f410 con 0x7fd9a80b1360 2024-03-21T08:43:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.643+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd990012a10 con 0x7fd9a80b1360 2024-03-21T08:43:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.643+0000 7fd99ea00640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 0x7fd984069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.643+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd990085d80 con 0x7fd9a80b1360 2024-03-21T08:43:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.643+0000 7fd9a5e00640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 0x7fd984069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.643+0000 7fd9a5e00640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 0x7fd984069fe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd9980af000 tx=0x7fd9980a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:03.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.647+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd990055e30 con 0x7fd9a80b1360 2024-03-21T08:43:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd9a80b43f0 con 0x7fd9a80b1360 2024-03-21T08:43:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd99ea00640 1 -- 172.21.15.28:0/168114482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+82 (secure 0 0 0) 0x7fd990059ed0 con 0x7fd9a80b1360 2024-03-21T08:43:03.849 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 msgr2=0x7fd984069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 0x7fd984069fe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd9980af000 tx=0x7fd9980a7070 comp rx=0 tx=0).stop 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 msgr2=0x7fd9a8159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd99000c490 tx=0x7fd99000c950 comp rx=0 tx=0).stop 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 shutdown_connections 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9a815d060 0x7fd9a81598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9a80b1360 0x7fd9a8159240 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd984067b20 0x7fd984069fe0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 --2- 172.21.15.28:0/168114482 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd9a809a290 0x7fd9a806d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:03.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 >> 172.21.15.28:0/168114482 conn(0x7fd9a80b3510 msgr2=0x7fd9a8073c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:03.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.847+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 shutdown_connections 2024-03-21T08:43:03.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:03.851+0000 7fd9ad200640 1 -- 172.21.15.28:0/168114482 wait complete. 2024-03-21T08:43:03.869 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:43:03.870 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:03.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:03.870 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42286sec preferred_lft 42286sec 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:03.882 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:43:03.883 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:03.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:03.884 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:43:03.944 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:43:03.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:03.947 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:43:04.000 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:43:04.001 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:43:04.001 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:43:04.001 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:43:04.001 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:43:04.001 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:43:04.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:43:04.002 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:43:04.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.092 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:43:04.102 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:43:04.102 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:43:04.102 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:04.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:43:04.166 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:43:04.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:04.167 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:04.215 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:04.215 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:04.216 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:04.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:04.328 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:04.380 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:04.381 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:04.382 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:04.383 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:04.383 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:04.383 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:04.383 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:04.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.384 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:04.435 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:43:04.435 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:43:04.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:04.505 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:04.557 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:04.558 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:04.559 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:04.560 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:04.560 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:04.560 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:04.560 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:04.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.561 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:04.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:43:04.575+0000 7f75522e2580 -1 init, newargv = 0x56363f5cdbb0 newargc=15 2024-03-21T08:43:04.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38546]: starting ceph client 2024-03-21T08:43:04.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38546]: starting fuse 2024-03-21T08:43:04.588 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:43:04.588 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:43:04.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:43:04.588 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:43:04.588 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:43:04.588 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:43:04.588 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:43:04.589 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:43:04.590 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:43:04.713 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.38546.asok 2024-03-21T08:43:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:43:04.724 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38546.asok 2024-03-21T08:43:04.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38546.asok status 2024-03-21T08:43:04.904 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "38546", 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:43:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5979, 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5979 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1309999198 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1309999198 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.5979 192.168.0.1:0/1309999198", 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1309999198", 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 19, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 42, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:43:04.907 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:43:04.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.921 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:43:04.921 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:04.930 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:43:04.930 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:04.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:04.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:04.990 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-21T08:43:05.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:05.046 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-21T08:43:05.105 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:05.105 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:05.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.335+0000 7f9627000640 1 Processor -- start 2024-03-21T08:43:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.335+0000 7f9627000640 1 -- start start 2024-03-21T08:43:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9627000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f962015c680 0x7f96200ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9627000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9627000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200ceaa0 con 0x7f962006b170 2024-03-21T08:43:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9627000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96200ced80 con 0x7f962015c680 2024-03-21T08:43:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9627000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200cf060 con 0x7f96200cfaa0 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f961fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f962015c680 0x7f96200ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:44978/0 (socket says 172.21.15.28:44978) 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 -- 172.21.15.28:0/469745788 learned_addr learned my addr 172.21.15.28:0/469745788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9625000640 1 --1- 172.21.15.28:0/469745788 >> v1:172.21.15.151:6789/0 conn(0x7f962006b170 0x7f962006b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56300/0 (socket says 172.21.15.28:56300) 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 -- 172.21.15.28:0/469745788 >> v2:172.21.15.151:3300/0 conn(0x7f962015c680 msgr2=0x7f96200ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 --2- 172.21.15.28:0/469745788 >> v2:172.21.15.151:3300/0 conn(0x7f962015c680 0x7f96200ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 -- 172.21.15.28:0/469745788 >> v1:172.21.15.151:6789/0 conn(0x7f962006b170 legacy=0x7f962006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.339+0000 7f9617e00640 1 -- 172.21.15.28:0/469745788 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200cf340 con 0x7f96200cfaa0 2024-03-21T08:43:05.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9617e00640 1 --2- 172.21.15.28:0/469745788 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9608009870 tx=0x7f960801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a488dbdf982f899 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f961f400640 1 -- 172.21.15.28:0/469745788 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96080b0070 con 0x7f96200cfaa0 2024-03-21T08:43:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f961f400640 1 -- 172.21.15.28:0/469745788 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96080ab600 con 0x7f96200cfaa0 2024-03-21T08:43:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f961f400640 1 -- 172.21.15.28:0/469745788 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96080ab9d0 con 0x7f96200cfaa0 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 -- 172.21.15.28:0/469745788 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 msgr2=0x7f962015ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 --2- 172.21.15.28:0/469745788 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9608009870 tx=0x7f960801c4d0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 -- 172.21.15.28:0/469745788 shutdown_connections 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 --2- 172.21.15.28:0/469745788 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f96200cfaa0 0x7f962015ca60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 --2- 172.21.15.28:0/469745788 >> v2:172.21.15.151:3300/0 conn(0x7f962015c680 0x7f96200ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.343+0000 7f9627000640 1 -- 172.21.15.28:0/469745788 >> 172.21.15.28:0/469745788 conn(0x7f9620099a30 msgr2=0x7f9620099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- 172.21.15.28:0/469745788 shutdown_connections 2024-03-21T08:43:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- 172.21.15.28:0/469745788 wait complete. 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 Processor -- start 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- start start 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 0x7f96201515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 0x7f96201498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200cd580 con 0x7f962015c680 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200cd400 con 0x7f96200cfaa0 2024-03-21T08:43:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9627000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9620168aa0 con 0x7f962006b170 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47484/0 (socket says 172.21.15.28:47484) 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 -- 172.21.15.28:0/3636179526 learned_addr learned my addr 172.21.15.28:0/3636179526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f961fe00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 0x7f96201515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9617e00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 0x7f96201498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 -- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 msgr2=0x7f96201515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 0x7f96201515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 -- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 msgr2=0x7f96201498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 0x7f96201498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9625000640 1 -- 172.21.15.28:0/3636179526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620151af0 con 0x7f962015c680 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f961fe00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 0x7f96201515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.347+0000 7f9617e00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 0x7f96201498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f9625000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f961800c730 tx=0x7f961800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618017070 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800ce40 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800a840 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96200c9430 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96200c99f0 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962004f410 con 0x7f962015c680 2024-03-21T08:43:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.351+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f961801a070 con 0x7f962015c680 2024-03-21T08:43:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.355+0000 7f961cc00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 0x7f95fc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.355+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9618085ba0 con 0x7f962015c680 2024-03-21T08:43:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.355+0000 7f961fe00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 0x7f95fc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.355+0000 7f961fe00640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 0x7f95fc069fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9610009b10 tx=0x7f9610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.355+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9618085e20 con 0x7f962015c680 2024-03-21T08:43:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.519+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f962015e6f0 con 0x7f962015c680 2024-03-21T08:43:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f961cc00640 1 -- 172.21.15.28:0/3636179526 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f9620151af0 con 0x7f962015c680 2024-03-21T08:43:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:05.524 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 msgr2=0x7f95fc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 0x7f95fc069fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9610009b10 tx=0x7f9610008040 comp rx=0 tx=0).stop 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 msgr2=0x7f962006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f961800c730 tx=0x7f961800cbf0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 shutdown_connections 2024-03-21T08:43:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962015c680 0x7f962006ebc0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f96200cfaa0 0x7f96201498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f95fc067b20 0x7f95fc069fe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 --2- 172.21.15.28:0/3636179526 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f962006b170 0x7f96201515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 >> 172.21.15.28:0/3636179526 conn(0x7f9620099a30 msgr2=0x7f9620073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 shutdown_connections 2024-03-21T08:43:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.523+0000 7f9627000640 1 -- 172.21.15.28:0/3636179526 wait complete. 2024-03-21T08:43:05.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:05.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:05.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 Processor -- start 2024-03-21T08:43:05.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 -- start start 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7feb600cebd0 0x7feb60166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6015fba0 con 0x7feb6015bd20 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6015fe80 con 0x7feb600b0e80 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.686+0000 7feb65000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6008bfd0 con 0x7feb600cebd0 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb6015bd20 0x7feb600b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46642/0 (socket says 172.21.15.28:46642) 2024-03-21T08:43:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5de00640 1 -- 172.21.15.28:0/4165906540 learned_addr learned my addr 172.21.15.28:0/4165906540 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5e800640 1 --2- 172.21.15.28:0/4165906540 >> v2:172.21.15.151:3300/0 conn(0x7feb600cebd0 0x7feb60166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536314880 0 0) 0x7feb6015fba0 con 0x7feb6015bd20 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb44003530 con 0x7feb6015bd20 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781153069 0 0) 0x7feb6015fe80 con 0x7feb600b0e80 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6015fba0 con 0x7feb600b0e80 2024-03-21T08:43:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923821373 0 0) 0x7feb44003530 con 0x7feb6015bd20 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 >> v2:172.21.15.151:3300/0 conn(0x7feb600cebd0 msgr2=0x7feb60166d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 --2- 172.21.15.28:0/4165906540 >> v2:172.21.15.151:3300/0 conn(0x7feb600cebd0 0x7feb60166d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 >> v1:172.21.15.151:6789/0 conn(0x7feb600b0e80 legacy=0x7feb600ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60160160 con 0x7feb6015bd20 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7feb50001680 con 0x7feb6015bd20 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 >> v1:172.21.15.28:6789/0 conn(0x7feb6015bd20 legacy=0x7feb600b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb44003b50 0x7feb44003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 0x7feb44004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb440051d0 0x7feb4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4400c020 con 0x7feb44004810 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4400c1d0 con 0x7feb440051d0 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4400c3d0 con 0x7feb44003b50 2024-03-21T08:43:05.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 0x7feb44004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5de00640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb44003b50 0x7feb44003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5e800640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb440051d0 0x7feb4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 -- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb44003b50 msgr2=0x7feb44003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb44003b50 0x7feb44003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 -- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb440051d0 msgr2=0x7feb4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb440051d0 0x7feb4400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb55400640 1 -- 172.21.15.28:0/4165906540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44003530 con 0x7feb44004810 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5e800640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb440051d0 0x7feb4400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:05.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.690+0000 7feb5e800640 1 -- 172.21.15.28:0/4165906540 reap_dead start 2024-03-21T08:43:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb55400640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 0x7feb44004c90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7feb480018d0 tx=0x7feb4801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e78e1748a5d2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48001520 con 0x7feb44004810 2024-03-21T08:43:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb480052f0 con 0x7feb44004810 2024-03-21T08:43:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb5d400640 1 -- 172.21.15.28:0/4165906540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb480a88f0 con 0x7feb44004810 2024-03-21T08:43:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 msgr2=0x7feb44004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 0x7feb44004c90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7feb480018d0 tx=0x7feb4801ec90 comp rx=0 tx=0).stop 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- 172.21.15.28:0/4165906540 shutdown_connections 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 --2- 172.21.15.28:0/4165906540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb44004810 0x7feb44004c90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- 172.21.15.28:0/4165906540 >> 172.21.15.28:0/4165906540 conn(0x7feb600d1ab0 msgr2=0x7feb600d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- 172.21.15.28:0/4165906540 shutdown_connections 2024-03-21T08:43:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- 172.21.15.28:0/4165906540 wait complete. 2024-03-21T08:43:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 Processor -- start 2024-03-21T08:43:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.694+0000 7feb65000640 1 -- start start 2024-03-21T08:43:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb44004810 0x7feb600ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb6015df90 0x7feb600c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb600cc6a0 con 0x7feb6015ac40 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb600cc520 con 0x7feb44004810 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6015c100 con 0x7feb6015df90 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47500/0 (socket says 172.21.15.28:47500) 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 -- 172.21.15.28:0/695424382 learned_addr learned my addr 172.21.15.28:0/695424382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 -- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb6015df90 msgr2=0x7feb600c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb6015df90 0x7feb600c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 -- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb44004810 msgr2=0x7feb600ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb44004810 0x7feb600ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 -- 172.21.15.28:0/695424382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600caca0 con 0x7feb6015ac40 2024-03-21T08:43:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb5e800640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feb44004790 tx=0x7feb58010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58010e30 con 0x7feb6015ac40 2024-03-21T08:43:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb580039a0 con 0x7feb6015ac40 2024-03-21T08:43:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58003d70 con 0x7feb6015ac40 2024-03-21T08:43:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb600c5da0 con 0x7feb6015ac40 2024-03-21T08:43:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6015e410 con 0x7feb6015ac40 2024-03-21T08:43:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb600b4280 con 0x7feb6015ac40 2024-03-21T08:43:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.698+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7feb5801e020 con 0x7feb6015ac40 2024-03-21T08:43:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.702+0000 7feb56a00640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 0x7feb3c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.702+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb58086880 con 0x7feb6015ac40 2024-03-21T08:43:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.702+0000 7feb5de00640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 0x7feb3c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:05.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.702+0000 7feb5de00640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 0x7feb3c06a010 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feb50001060 tx=0x7feb50001e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:05.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.710+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb58056f90 con 0x7feb6015ac40 2024-03-21T08:43:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb6004f4f0 con 0x7feb6015ac40 2024-03-21T08:43:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb56a00640 1 -- 172.21.15.28:0/695424382 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb58056890 con 0x7feb6015ac40 2024-03-21T08:43:05.876 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 msgr2=0x7feb3c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 0x7feb3c06a010 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feb50001060 tx=0x7feb50001e40 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 msgr2=0x7feb6015b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feb44004790 tx=0x7feb58010590 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 shutdown_connections 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb6015ac40 0x7feb6015b0c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb3c067b50 0x7feb3c06a010 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb6015df90 0x7feb600c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 --2- 172.21.15.28:0/695424382 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb44004810 0x7feb600ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 >> 172.21.15.28:0/695424382 conn(0x7feb600d1ab0 msgr2=0x7feb60073d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 shutdown_connections 2024-03-21T08:43:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:05.874+0000 7feb65000640 1 -- 172.21.15.28:0/695424382 wait complete. 2024-03-21T08:43:05.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:06.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 Processor -- start 2024-03-21T08:43:06.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 -- start start 2024-03-21T08:43:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3f415aa30 0x7fa3f415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f40ce590 0x7fa3f40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f415be50 con 0x7fa3f415b350 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f40cf930 con 0x7fa3f415aa30 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3fa400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f415b890 con 0x7fa3f40ce590 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3f415aa30 0x7fa3f415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa3f415b350 0x7fa3f40cd720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46644/0 (socket says 172.21.15.28:46644) 2024-03-21T08:43:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2947698966 learned_addr learned my addr 172.21.15.28:0/2947698966 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3ebe00640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f40ce590 0x7fa3f40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3400640 1 -- 172.21.15.28:0/2947698966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f40ce590 msgr2=0x7fa3f40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3400640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f40ce590 0x7fa3f40d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3400640 1 -- 172.21.15.28:0/2947698966 >> v1:172.21.15.28:6789/0 conn(0x7fa3f415b350 legacy=0x7fa3f40cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f3400640 1 -- 172.21.15.28:0/2947698966 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f415bb70 con 0x7fa3f415aa30 2024-03-21T08:43:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.042+0000 7fa3f2a00640 1 -- 172.21.15.28:0/2947698966 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653946158 0 0) 0x7fa3f415be50 con 0x7fa3f415b350 2024-03-21T08:43:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3f3400640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f415aa30 0x7fa3f415ae10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40097c0 tx=0x7fa3e401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5bccd6425e18f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3f2a00640 1 -- 172.21.15.28:0/2947698966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40ab600 con 0x7fa3f415aa30 2024-03-21T08:43:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3f2a00640 1 -- 172.21.15.28:0/2947698966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e40b0070 con 0x7fa3f415aa30 2024-03-21T08:43:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3f2a00640 1 -- 172.21.15.28:0/2947698966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40ab9c0 con 0x7fa3f415aa30 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 -- 172.21.15.28:0/2947698966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f415aa30 msgr2=0x7fa3f415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f415aa30 0x7fa3f415ae10 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40097c0 tx=0x7fa3e401c3e0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 -- 172.21.15.28:0/2947698966 shutdown_connections 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f40ce590 0x7fa3f40d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 --2- 172.21.15.28:0/2947698966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f415aa30 0x7fa3f415ae10 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 -- 172.21.15.28:0/2947698966 >> 172.21.15.28:0/2947698966 conn(0x7fa3f4099a30 msgr2=0x7fa3f4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 -- 172.21.15.28:0/2947698966 shutdown_connections 2024-03-21T08:43:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.046+0000 7fa3fa400640 1 -- 172.21.15.28:0/2947698966 wait complete. 2024-03-21T08:43:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 Processor -- start 2024-03-21T08:43:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- start start 2024-03-21T08:43:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f415aa30 0x7fa3f40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f415b350 0x7fa3f40aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40cbd80 con 0x7fa3f40ce590 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40cbc00 con 0x7fa3f415b350 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40cbf00 con 0x7fa3f415aa30 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47506/0 (socket says 172.21.15.28:47506) 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 -- 172.21.15.28:0/234413018 learned_addr learned my addr 172.21.15.28:0/234413018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3ebe00640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f415aa30 0x7fa3f40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3e00640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f415b350 0x7fa3f40aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 -- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f415aa30 msgr2=0x7fa3f40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f415aa30 0x7fa3f40ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 -- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f415b350 msgr2=0x7fa3f40aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f415b350 0x7fa3f40aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 -- 172.21.15.28:0/234413018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f40c90e0 con 0x7fa3f40ce590 2024-03-21T08:43:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3f3400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40097c0 tx=0x7fa3e401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40bf610 con 0x7fa3f40ce590 2024-03-21T08:43:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e40bf7d0 con 0x7fa3f40ce590 2024-03-21T08:43:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f415c070 con 0x7fa3f40ce590 2024-03-21T08:43:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e40bfbc0 con 0x7fa3f40ce590 2024-03-21T08:43:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.050+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f40b4dc0 con 0x7fa3f40ce590 2024-03-21T08:43:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f409ea60 con 0x7fa3f40ce590 2024-03-21T08:43:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa3e40bd020 con 0x7fa3f40ce590 2024-03-21T08:43:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3eb400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 0x7fa3d006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa3e4120a30 con 0x7fa3f40ce590 2024-03-21T08:43:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3ebe00640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 0x7fa3d006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.054+0000 7fa3ebe00640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 0x7fa3d006a010 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40b3a80 tx=0x7fa3dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.058+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa3e4016090 con 0x7fa3f40ce590 2024-03-21T08:43:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3f406ed60 con 0x7fa3f40ce590 2024-03-21T08:43:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3eb400640 1 -- 172.21.15.28:0/234413018 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3e40f1130 con 0x7fa3f40ce590 2024-03-21T08:43:06.228 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 msgr2=0x7fa3d006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 0x7fa3d006a010 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40b3a80 tx=0x7fa3dc008040 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 msgr2=0x7fa3f40b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40097c0 tx=0x7fa3e401ac20 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 shutdown_connections 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa3f415b350 0x7fa3f40aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa3d0067b50 0x7fa3d006a010 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa3f415aa30 0x7fa3f40ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 --2- 172.21.15.28:0/234413018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f40ce590 0x7fa3f40b5ec0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 >> 172.21.15.28:0/234413018 conn(0x7fa3f4099a30 msgr2=0x7fa3f40cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 shutdown_connections 2024-03-21T08:43:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.226+0000 7fa3fa400640 1 -- 172.21.15.28:0/234413018 wait complete. 2024-03-21T08:43:06.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 Processor -- start 2024-03-21T08:43:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 -- start start 2024-03-21T08:43:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 0x7f16f00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f006d070 con 0x7f16f006cbe0 2024-03-21T08:43:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f00d05f0 con 0x7f16f015e1e0 2024-03-21T08:43:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f7600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00b3010 con 0x7f16f00d0190 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16e7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f4c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 0x7f16f00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16e7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:45024/0 (socket says 172.21.15.28:45024) 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16e7e00640 1 -- 172.21.15.28:0/1500059361 learned_addr learned my addr 172.21.15.28:0/1500059361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.390+0000 7f16f5600640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f16f006cbe0 0x7f16f0161c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56326/0 (socket says 172.21.15.28:56326) 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16e7e00640 1 -- 172.21.15.28:0/1500059361 >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 msgr2=0x7f16f00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16efe00640 1 -- 172.21.15.28:0/1500059361 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699315196 0 0) 0x7f16f006d070 con 0x7f16f006cbe0 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16e7e00640 1 --2- 172.21.15.28:0/1500059361 >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 0x7f16f00cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16e7e00640 1 -- 172.21.15.28:0/1500059361 >> v1:172.21.15.151:6789/0 conn(0x7f16f006cbe0 legacy=0x7f16f0161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16e7e00640 1 -- 172.21.15.28:0/1500059361 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f006d310 con 0x7f16f00d0190 2024-03-21T08:43:06.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16f4c00640 1 --2- 172.21.15.28:0/1500059361 >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 0x7f16f00cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16e7e00640 1 --2- 172.21.15.28:0/1500059361 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009870 tx=0x7f16d801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ad2dcbe47eabebd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16efe00640 1 -- 172.21.15.28:0/1500059361 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80ab650 con 0x7f16f00d0190 2024-03-21T08:43:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16efe00640 1 -- 172.21.15.28:0/1500059361 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d80b0070 con 0x7f16f00d0190 2024-03-21T08:43:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.394+0000 7f16efe00640 1 -- 172.21.15.28:0/1500059361 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80aba10 con 0x7f16f00d0190 2024-03-21T08:43:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- 172.21.15.28:0/1500059361 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 msgr2=0x7f16f006c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- 172.21.15.28:0/1500059361 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009870 tx=0x7f16d801c560 comp rx=0 tx=0).stop 2024-03-21T08:43:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- 172.21.15.28:0/1500059361 shutdown_connections 2024-03-21T08:43:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- 172.21.15.28:0/1500059361 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f00d0190 0x7f16f006c6a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- 172.21.15.28:0/1500059361 >> v2:172.21.15.151:3300/0 conn(0x7f16f015e1e0 0x7f16f00cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- 172.21.15.28:0/1500059361 >> 172.21.15.28:0/1500059361 conn(0x7f16f00d1ab0 msgr2=0x7f16f00d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- 172.21.15.28:0/1500059361 shutdown_connections 2024-03-21T08:43:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- 172.21.15.28:0/1500059361 wait complete. 2024-03-21T08:43:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 Processor -- start 2024-03-21T08:43:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- start start 2024-03-21T08:43:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f006cbe0 0x7f16f014a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f015e1e0 0x7f16f014a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f016b0c0 con 0x7f16f00d0190 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f016af40 con 0x7f16f015e1e0 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f7600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f016b240 con 0x7f16f006cbe0 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47520/0 (socket says 172.21.15.28:47520) 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 -- 172.21.15.28:0/2724877616 learned_addr learned my addr 172.21.15.28:0/2724877616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f4c00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f006cbe0 0x7f16f014a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16f5600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f015e1e0 0x7f16f014a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 -- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f006cbe0 msgr2=0x7f16f014a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f006cbe0 0x7f16f014a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 -- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f015e1e0 msgr2=0x7f16f014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f015e1e0 0x7f16f014a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.398+0000 7f16e7e00640 1 -- 172.21.15.28:0/2724877616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f01542a0 con 0x7f16f00d0190 2024-03-21T08:43:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16e7e00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f16d801a190 tx=0x7f16d801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80ba070 con 0x7f16f00d0190 2024-03-21T08:43:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8004230 con 0x7f16f00d0190 2024-03-21T08:43:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80ab8a0 con 0x7f16f00d0190 2024-03-21T08:43:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f015c4e0 con 0x7f16f00d0190 2024-03-21T08:43:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f015caa0 con 0x7f16f00d0190 2024-03-21T08:43:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f004f4f0 con 0x7f16f00d0190 2024-03-21T08:43:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f16d80aba40 con 0x7f16f00d0190 2024-03-21T08:43:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 0x7f16cc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f16d8120380 con 0x7f16f00d0190 2024-03-21T08:43:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.402+0000 7f16f4c00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 0x7f16cc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.406+0000 7f16f4c00640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 0x7f16cc069fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009580 tx=0x7f16e0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.406+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f16f01542a0 con 0x7f16f00d0190 2024-03-21T08:43:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.570+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16f008bd40 con 0x7f16f00d0190 2024-03-21T08:43:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.570+0000 7f16ed600640 1 -- 172.21.15.28:0/2724877616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16d8016150 con 0x7f16f00d0190 2024-03-21T08:43:06.573 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 msgr2=0x7f16cc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 0x7f16cc069fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009580 tx=0x7f16e0009190 comp rx=0 tx=0).stop 2024-03-21T08:43:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 msgr2=0x7f16f0149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f16d801a190 tx=0x7f16d801ada0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 shutdown_connections 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f16f015e1e0 0x7f16f014a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f00d0190 0x7f16f0149ac0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f16cc067b20 0x7f16cc069fe0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 --2- 172.21.15.28:0/2724877616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f16f006cbe0 0x7f16f014a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 >> 172.21.15.28:0/2724877616 conn(0x7f16f00d1ab0 msgr2=0x7f16f00cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 shutdown_connections 2024-03-21T08:43:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.574+0000 7f16f7600640 1 -- 172.21.15.28:0/2724877616 wait complete. 2024-03-21T08:43:06.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:43:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 Processor -- start 2024-03-21T08:43:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 -- start start 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f801015bdc0 0x7f80100cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80100cc640 0x7f80100d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80100ccbd0 con 0x7f80100d08f0 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100cd7e0 con 0x7f80100cc640 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8017400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100cceb0 con 0x7f801015bdc0 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f800fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80100cc640 0x7f80100d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f800fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80100cc640 0x7f80100d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47532/0 (socket says 172.21.15.28:47532) 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f800fe00640 1 -- 172.21.15.28:0/1658698198 learned_addr learned my addr 172.21.15.28:0/1658698198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.738+0000 7f8014a00640 1 --2- 172.21.15.28:0/1658698198 >> v2:172.21.15.151:3300/0 conn(0x7f801015bdc0 0x7f80100cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800fe00640 1 -- 172.21.15.28:0/1658698198 >> v2:172.21.15.151:3300/0 conn(0x7f801015bdc0 msgr2=0x7f80100cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800fe00640 1 --2- 172.21.15.28:0/1658698198 >> v2:172.21.15.151:3300/0 conn(0x7f801015bdc0 0x7f80100cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800fe00640 1 -- 172.21.15.28:0/1658698198 >> v1:172.21.15.151:6789/0 conn(0x7f80100d08f0 legacy=0x7f8010099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800fe00640 1 -- 172.21.15.28:0/1658698198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100cd190 con 0x7f80100cc640 2024-03-21T08:43:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800fe00640 1 --2- 172.21.15.28:0/1658698198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f80100d03b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f80100726c0 tx=0x7f7ffc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872f3acfbec184b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800f400640 1 -- 172.21.15.28:0/1658698198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0ab620 con 0x7f80100cc640 2024-03-21T08:43:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800f400640 1 -- 172.21.15.28:0/1658698198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc0b0070 con 0x7f80100cc640 2024-03-21T08:43:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f800f400640 1 -- 172.21.15.28:0/1658698198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0ab9e0 con 0x7f80100cc640 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 -- 172.21.15.28:0/1658698198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 msgr2=0x7f80100d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 --2- 172.21.15.28:0/1658698198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f80100d03b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f80100726c0 tx=0x7f7ffc01c460 comp rx=0 tx=0).stop 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 -- 172.21.15.28:0/1658698198 shutdown_connections 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 --2- 172.21.15.28:0/1658698198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f80100d03b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 --2- 172.21.15.28:0/1658698198 >> v2:172.21.15.151:3300/0 conn(0x7f801015bdc0 0x7f80100cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 -- 172.21.15.28:0/1658698198 >> 172.21.15.28:0/1658698198 conn(0x7f80100b3620 msgr2=0x7f80100b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 -- 172.21.15.28:0/1658698198 shutdown_connections 2024-03-21T08:43:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.742+0000 7f8017400640 1 -- 172.21.15.28:0/1658698198 wait complete. 2024-03-21T08:43:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 Processor -- start 2024-03-21T08:43:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 -- start start 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 0x7f801015ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801015bdc0 0x7f801015a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80100b1b40 con 0x7f80100cc640 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801009a5b0 con 0x7f801015bdc0 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80100b1cc0 con 0x7f80100d08f0 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47536/0 (socket says 172.21.15.28:47536) 2024-03-21T08:43:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 -- 172.21.15.28:0/1649391569 learned_addr learned my addr 172.21.15.28:0/1649391569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f800fe00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 0x7f801015ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8015400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801015bdc0 0x7f801015a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 -- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 msgr2=0x7f801015ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 0x7f801015ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 -- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801015bdc0 msgr2=0x7f801015a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801015bdc0 0x7f801015a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 -- 172.21.15.28:0/1649391569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80101592d0 con 0x7f80100cc640 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f800fe00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 0x7f801015ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8014a00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f800400d620 tx=0x7f800400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400b620 con 0x7f80100cc640 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004019070 con 0x7f80100cc640 2024-03-21T08:43:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400b9e0 con 0x7f80100cc640 2024-03-21T08:43:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.746+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80101595b0 con 0x7f80100cc640 2024-03-21T08:43:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.750+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8010159b70 con 0x7f80100cc640 2024-03-21T08:43:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.750+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801009faf0 con 0x7f80100cc640 2024-03-21T08:43:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.750+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f800400bb80 con 0x7f80100cc640 2024-03-21T08:43:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.750+0000 7f800cc00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 0x7f7ff006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.750+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f80040865a0 con 0x7f80100cc640 2024-03-21T08:43:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.754+0000 7f800fe00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 0x7f7ff006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.754+0000 7f800fe00640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 0x7f7ff006a010 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0af000 tx=0x7f7ffc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.754+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8004056c20 con 0x7f80100cc640 2024-03-21T08:43:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.918+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f80100b4490 con 0x7f80100cc640 2024-03-21T08:43:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.918+0000 7f800cc00640 1 -- 172.21.15.28:0/1649391569 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7f8004056520 con 0x7f80100cc640 2024-03-21T08:43:06.923 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 msgr2=0x7f7ff006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 0x7f7ff006a010 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0af000 tx=0x7f7ffc0a7000 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 msgr2=0x7f8010154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f800400d620 tx=0x7f800400dae0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 shutdown_connections 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f801015bdc0 0x7f801015a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ff0067b50 0x7f7ff006a010 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80100d08f0 0x7f801015ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 --2- 172.21.15.28:0/1649391569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100cc640 0x7f8010154820 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 >> 172.21.15.28:0/1649391569 conn(0x7f80100b3620 msgr2=0x7f8010071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 shutdown_connections 2024-03-21T08:43:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:06.922+0000 7f8017400640 1 -- 172.21.15.28:0/1649391569 wait complete. 2024-03-21T08:43:06.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 Processor -- start 2024-03-21T08:43:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 -- start start 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f7d880d0360 0x7f7d8806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8806cbe0 0x7f7d880cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880b3010 con 0x7f7d8815e1e0 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8806d310 con 0x7f7d8806cbe0 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8f000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8806d5f0 con 0x7f7d880d0360 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8806cbe0 0x7f7d880cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8806cbe0 0x7f7d880cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47538/0 (socket says 172.21.15.28:47538) 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 -- 172.21.15.28:0/3809945593 learned_addr learned my addr 172.21.15.28:0/3809945593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d7fe00640 1 --2- 172.21.15.28:0/3809945593 >> v2:172.21.15.151:3300/0 conn(0x7f7d880d0360 0x7f7d8806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 -- 172.21.15.28:0/3809945593 >> v2:172.21.15.151:3300/0 conn(0x7f7d880d0360 msgr2=0x7f7d8806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 --2- 172.21.15.28:0/3809945593 >> v2:172.21.15.151:3300/0 conn(0x7f7d880d0360 0x7f7d8806c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 -- 172.21.15.28:0/3809945593 >> v1:172.21.15.151:6789/0 conn(0x7f7d8815e1e0 legacy=0x7f7d880cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.086+0000 7f7d8d000640 1 -- 172.21.15.28:0/3809945593 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8806d8d0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8d000640 1 --2- 172.21.15.28:0/3809945593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d880cc4e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7d880d0d20 tx=0x7f7d8001d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0893962a037905 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d87400640 1 -- 172.21.15.28:0/3809945593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d800a57d0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d87400640 1 -- 172.21.15.28:0/3809945593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d800a5970 con 0x7f7d8806cbe0 2024-03-21T08:43:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d87400640 1 -- 172.21.15.28:0/3809945593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d800a5d40 con 0x7f7d8806cbe0 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 -- 172.21.15.28:0/3809945593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 msgr2=0x7f7d880cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 --2- 172.21.15.28:0/3809945593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d880cc4e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7d880d0d20 tx=0x7f7d8001d450 comp rx=0 tx=0).stop 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 -- 172.21.15.28:0/3809945593 shutdown_connections 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 --2- 172.21.15.28:0/3809945593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d880cc4e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 --2- 172.21.15.28:0/3809945593 >> v2:172.21.15.151:3300/0 conn(0x7f7d880d0360 0x7f7d8806c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 -- 172.21.15.28:0/3809945593 >> 172.21.15.28:0/3809945593 conn(0x7f7d880d1ab0 msgr2=0x7f7d880d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 -- 172.21.15.28:0/3809945593 shutdown_connections 2024-03-21T08:43:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.090+0000 7f7d8f000640 1 -- 172.21.15.28:0/3809945593 wait complete. 2024-03-21T08:43:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 Processor -- start 2024-03-21T08:43:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- start start 2024-03-21T08:43:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d880d0360 0x7f7d88149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d8815e1e0 0x7f7d880c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cd5c0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cd440 con 0x7f7d880d0360 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cd740 con 0x7f7d8815e1e0 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47554/0 (socket says 172.21.15.28:47554) 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 -- 172.21.15.28:0/2313904968 learned_addr learned my addr 172.21.15.28:0/2313904968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d7fe00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d880d0360 0x7f7d88149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8d000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d8815e1e0 0x7f7d880c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 -- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d8815e1e0 msgr2=0x7f7d880c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d8815e1e0 0x7f7d880c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 -- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d880d0360 msgr2=0x7f7d88149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d880d0360 0x7f7d88149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 -- 172.21.15.28:0/2313904968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8814a4d0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800c420 tx=0x7f7d7800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d780052d0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78005470 con 0x7f7d8806cbe0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d78007260 con 0x7f7d8806cbe0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8814a7b0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8814ad70 con 0x7f7d8806cbe0 2024-03-21T08:43:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.094+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88067200 con 0x7f7d8806cbe0 2024-03-21T08:43:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.098+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7d78007400 con 0x7f7d8806cbe0 2024-03-21T08:43:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.098+0000 7f7d84c00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 0x7f7d64069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.098+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7d78085690 con 0x7f7d8806cbe0 2024-03-21T08:43:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.098+0000 7f7d7fe00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 0x7f7d64069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.098+0000 7f7d7fe00640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 0x7f7d64069fe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7d881581c0 tx=0x7f7d70009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.106+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7d780556c0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7d8804f4f0 con 0x7f7d8806cbe0 2024-03-21T08:43:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d84c00640 1 -- 172.21.15.28:0/2313904968 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7d78059760 con 0x7f7d8806cbe0 2024-03-21T08:43:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 msgr2=0x7f7d64069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 0x7f7d64069fe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7d881581c0 tx=0x7f7d70009190 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 msgr2=0x7f7d8814bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800c420 tx=0x7f7d7800c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 shutdown_connections 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7d64067b20 0x7f7d64069fe0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7d8815e1e0 0x7f7d880c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7d880d0360 0x7f7d88149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 --2- 172.21.15.28:0/2313904968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8806cbe0 0x7f7d8814bc80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 >> 172.21.15.28:0/2313904968 conn(0x7f7d880d1ab0 msgr2=0x7f7d8808b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 shutdown_connections 2024-03-21T08:43:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.282+0000 7f7d8f000640 1 -- 172.21.15.28:0/2313904968 wait complete. 2024-03-21T08:43:07.314 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:07.314 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:07.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 Processor -- start 2024-03-21T08:43:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 -- start start 2024-03-21T08:43:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 0x7f3c2006b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3c2006eb30 0x7f3c2006b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c200d0540 con 0x7f3c2015dd00 2024-03-21T08:43:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c200cd6e0 con 0x7f3c2006eb30 2024-03-21T08:43:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c24a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200cd9c0 con 0x7f3c200aff70 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c17e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 0x7f3c2006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c1de00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f3c2015dd00 0x7f3c200cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56364/0 (socket says 172.21.15.28:56364) 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c1de00640 1 -- 172.21.15.28:0/2674483887 learned_addr learned my addr 172.21.15.28:0/2674483887 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c1e800640 1 --2- 172.21.15.28:0/2674483887 >> v2:172.21.15.151:3300/0 conn(0x7f3c2006eb30 0x7f3c2006b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c17e00640 1 -- 172.21.15.28:0/2674483887 >> v2:172.21.15.151:3300/0 conn(0x7f3c2006eb30 msgr2=0x7f3c2006b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c17e00640 1 --2- 172.21.15.28:0/2674483887 >> v2:172.21.15.151:3300/0 conn(0x7f3c2006eb30 0x7f3c2006b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c17e00640 1 -- 172.21.15.28:0/2674483887 >> v1:172.21.15.151:6789/0 conn(0x7f3c2015dd00 legacy=0x7f3c200cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c17e00640 1 -- 172.21.15.28:0/2674483887 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c200cdca0 con 0x7f3c200aff70 2024-03-21T08:43:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.506+0000 7f3c1d400640 1 -- 172.21.15.28:0/2674483887 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372587495 0 0) 0x7f3c200d0540 con 0x7f3c2015dd00 2024-03-21T08:43:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c17e00640 1 --2- 172.21.15.28:0/2674483887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 0x7f3c2006b170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c080097c0 tx=0x7f3c0801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d4fd24851b1d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c1d400640 1 -- 172.21.15.28:0/2674483887 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080b1070 con 0x7f3c200aff70 2024-03-21T08:43:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c1d400640 1 -- 172.21.15.28:0/2674483887 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c080ac620 con 0x7f3c200aff70 2024-03-21T08:43:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c1d400640 1 -- 172.21.15.28:0/2674483887 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080ac9f0 con 0x7f3c200aff70 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 -- 172.21.15.28:0/2674483887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 msgr2=0x7f3c2006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 --2- 172.21.15.28:0/2674483887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 0x7f3c2006b170 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c080097c0 tx=0x7f3c0801c470 comp rx=0 tx=0).stop 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 -- 172.21.15.28:0/2674483887 shutdown_connections 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 --2- 172.21.15.28:0/2674483887 >> v2:172.21.15.151:3300/0 conn(0x7f3c2006eb30 0x7f3c2006b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 --2- 172.21.15.28:0/2674483887 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c200aff70 0x7f3c2006b170 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 -- 172.21.15.28:0/2674483887 >> 172.21.15.28:0/2674483887 conn(0x7f3c20099a30 msgr2=0x7f3c20099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 -- 172.21.15.28:0/2674483887 shutdown_connections 2024-03-21T08:43:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.510+0000 7f3c24a00640 1 -- 172.21.15.28:0/2674483887 wait complete. 2024-03-21T08:43:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 Processor -- start 2024-03-21T08:43:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 -- start start 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c2006eb30 0x7f3c2015f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c2015dd00 0x7f3c20154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201623c0 con 0x7f3c200aff70 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20162240 con 0x7f3c2015dd00 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c24a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c20162540 con 0x7f3c2006eb30 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47558/0 (socket says 172.21.15.28:47558) 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 -- 172.21.15.28:0/1783273204 learned_addr learned my addr 172.21.15.28:0/1783273204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c1de00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c2006eb30 0x7f3c2015f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c1e800640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c2015dd00 0x7f3c20154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 -- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c2006eb30 msgr2=0x7f3c2015f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c2006eb30 0x7f3c2015f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 -- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c2015dd00 msgr2=0x7f3c20154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c2015dd00 0x7f3c20154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.514+0000 7f3c17e00640 1 -- 172.21.15.28:0/1783273204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2015fb00 con 0x7f3c200aff70 2024-03-21T08:43:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c17e00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3c080af040 tx=0x7f3c08019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080b1040 con 0x7f3c200aff70 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c080bb070 con 0x7f3c200aff70 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080b5670 con 0x7f3c200aff70 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c24a00640 1 -- 172.21.15.28:0/1783273204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c201563a0 con 0x7f3c200aff70 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c24a00640 1 -- 172.21.15.28:0/1783273204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c20156960 con 0x7f3c200aff70 2024-03-21T08:43:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2004f410 con 0x7f3c200aff70 2024-03-21T08:43:07.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.518+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f3c080b5810 con 0x7f3c200aff70 2024-03-21T08:43:07.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.522+0000 7f3c16000640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 0x7f3bfc06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.522+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3c0811fe70 con 0x7f3c200aff70 2024-03-21T08:43:07.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.522+0000 7f3c1de00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 0x7f3bfc06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.522+0000 7f3c1de00640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 0x7f3bfc06a010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3c10000f70 tx=0x7f3c10007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.522+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3c08016090 con 0x7f3c200aff70 2024-03-21T08:43:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.686+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c200ccfe0 con 0x7f3c200aff70 2024-03-21T08:43:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c16000640 1 -- 172.21.15.28:0/1783273204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f3c080eff20 con 0x7f3c200aff70 2024-03-21T08:43:07.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:07.692 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 msgr2=0x7f3bfc06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 0x7f3bfc06a010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3c10000f70 tx=0x7f3c10007df0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 msgr2=0x7f3c20157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3c080af040 tx=0x7f3c08019210 comp rx=0 tx=0).stop 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 shutdown_connections 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f3c2015dd00 0x7f3c20154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200aff70 0x7f3c20157440 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3bfc067b50 0x7f3bfc06a010 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 --2- 172.21.15.28:0/1783273204 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3c2006eb30 0x7f3c2015f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 >> 172.21.15.28:0/1783273204 conn(0x7f3c20099a30 msgr2=0x7f3c200af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 shutdown_connections 2024-03-21T08:43:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.690+0000 7f3c0f400640 1 -- 172.21.15.28:0/1783273204 wait complete. 2024-03-21T08:43:07.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:07.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 Processor -- start 2024-03-21T08:43:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 -- start start 2024-03-21T08:43:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f24c415bc90 0x7f24c408bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c40cec40 0x7f24c415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c408c080 con 0x7f24c415f890 2024-03-21T08:43:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c40cfe80 con 0x7f24c415bc90 2024-03-21T08:43:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c8c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c4160400 con 0x7f24c40cec40 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24bbe00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f24c415f890 0x7f24c40ce530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56380/0 (socket says 172.21.15.28:56380) 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24bbe00640 1 -- 172.21.15.28:0/38376827 learned_addr learned my addr 172.21.15.28:0/38376827 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c2800640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c40cec40 0x7f24c415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1e00640 1 --2- 172.21.15.28:0/38376827 >> v2:172.21.15.151:3300/0 conn(0x7f24c415bc90 0x7f24c408bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1400640 1 -- 172.21.15.28:0/38376827 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535210557 0 0) 0x7f24c408c080 con 0x7f24c415f890 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1400640 1 -- 172.21.15.28:0/38376827 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a8003530 con 0x7f24c415f890 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1e00640 1 -- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c40cec40 msgr2=0x7f24c415fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1e00640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c40cec40 0x7f24c415fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1e00640 1 -- 172.21.15.28:0/38376827 >> v1:172.21.15.151:6789/0 conn(0x7f24c415f890 legacy=0x7f24c40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.858+0000 7f24c1e00640 1 -- 172.21.15.28:0/38376827 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c408b060 con 0x7f24c415bc90 2024-03-21T08:43:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c1e00640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c415bc90 0x7f24c408bb40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f24b40097c0 tx=0x7f24b401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3244aa4eab001d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c1400640 1 -- 172.21.15.28:0/38376827 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b40b5070 con 0x7f24c415bc90 2024-03-21T08:43:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c1400640 1 -- 172.21.15.28:0/38376827 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b40a8990 con 0x7f24c415bc90 2024-03-21T08:43:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c1400640 1 -- 172.21.15.28:0/38376827 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b40a8d60 con 0x7f24c415bc90 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 -- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c415bc90 msgr2=0x7f24c408bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c415bc90 0x7f24c408bb40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f24b40097c0 tx=0x7f24b401ec90 comp rx=0 tx=0).stop 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 -- 172.21.15.28:0/38376827 shutdown_connections 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c40cec40 0x7f24c415fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 --2- 172.21.15.28:0/38376827 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c415bc90 0x7f24c408bb40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 -- 172.21.15.28:0/38376827 >> 172.21.15.28:0/38376827 conn(0x7f24c40d1b20 msgr2=0x7f24c40d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 -- 172.21.15.28:0/38376827 shutdown_connections 2024-03-21T08:43:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.862+0000 7f24c8c00640 1 -- 172.21.15.28:0/38376827 wait complete. 2024-03-21T08:43:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 Processor -- start 2024-03-21T08:43:07.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- start start 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c40cec40 0x7f24c41e6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c415f890 0x7f24c41e5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c41606e0 con 0x7f24c415bc90 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c408b620 con 0x7f24c40cec40 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c415c070 con 0x7f24c415f890 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47562/0 (socket says 172.21.15.28:47562) 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 -- 172.21.15.28:0/4144021004 learned_addr learned my addr 172.21.15.28:0/4144021004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c2800640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c415f890 0x7f24c41e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c1e00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c40cec40 0x7f24c41e6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 -- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c415f890 msgr2=0x7f24c41e5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c415f890 0x7f24c41e5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 -- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c40cec40 msgr2=0x7f24c41e6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c40cec40 0x7f24c41e6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 -- 172.21.15.28:0/4144021004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c406cc60 con 0x7f24c415bc90 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24bbe00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00c420 tx=0x7f24ac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac017070 con 0x7f24c415bc90 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac005cf0 con 0x7f24c415bc90 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0072d0 con 0x7f24c415bc90 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c40c7c80 con 0x7f24c415bc90 2024-03-21T08:43:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.866+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c406d1c0 con 0x7f24c415bc90 2024-03-21T08:43:07.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c404f4f0 con 0x7f24c415bc90 2024-03-21T08:43:07.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f24ac01a070 con 0x7f24c415bc90 2024-03-21T08:43:07.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24ba000640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 0x7f24a006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:07.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f24ac0855b0 con 0x7f24c415bc90 2024-03-21T08:43:07.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24c1e00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 0x7f24a006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:07.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.870+0000 7f24c1e00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 0x7f24a006a010 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f24b40b2000 tx=0x7f24b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:07.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:07.874+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f24c406cc60 con 0x7f24c415bc90 2024-03-21T08:43:07.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:43:07.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:43:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.038+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24c40b33f0 con 0x7f24c415bc90 2024-03-21T08:43:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.038+0000 7f24ba000640 1 -- 172.21.15.28:0/4144021004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24ac055550 con 0x7f24c415bc90 2024-03-21T08:43:08.042 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 msgr2=0x7f24a006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 0x7f24a006a010 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f24b40b2000 tx=0x7f24b4008040 comp rx=0 tx=0).stop 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 msgr2=0x7f24c4154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00c420 tx=0x7f24ac00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 shutdown_connections 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f24a0067b50 0x7f24a006a010 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f24c415f890 0x7f24c41e5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c415bc90 0x7f24c4154f70 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 --2- 172.21.15.28:0/4144021004 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f24c40cec40 0x7f24c41e6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 >> 172.21.15.28:0/4144021004 conn(0x7f24c40d1b20 msgr2=0x7f24c40d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 shutdown_connections 2024-03-21T08:43:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.042+0000 7f24c8c00640 1 -- 172.21.15.28:0/4144021004 wait complete. 2024-03-21T08:43:08.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 Processor -- start 2024-03-21T08:43:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 -- start start 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7670071a80 0x7f767006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76700cde40 0x7f7670071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76700723a0 con 0x7f767015cbb0 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767015c720 con 0x7f76700cde40 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.246+0000 7f7677800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76700ce5b0 con 0x7f7670071a80 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76700cde40 0x7f7670071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76700cde40 0x7f7670071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47576/0 (socket says 172.21.15.28:47576) 2024-03-21T08:43:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 -- 172.21.15.28:0/227941626 learned_addr learned my addr 172.21.15.28:0/227941626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f7674e00640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7670071a80 0x7f767006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 -- 172.21.15.28:0/227941626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7670071a80 msgr2=0x7f767006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7670071a80 0x7f767006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 -- 172.21.15.28:0/227941626 >> v1:172.21.15.151:6789/0 conn(0x7f767015cbb0 legacy=0x7f76700cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 -- 172.21.15.28:0/227941626 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700ce890 con 0x7f76700cde40 2024-03-21T08:43:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766fe00640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76700cde40 0x7f7670071e60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f765c009870 tx=0x7f765c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1a3d45c0f1ffc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766f400640 1 -- 172.21.15.28:0/227941626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0aa620 con 0x7f76700cde40 2024-03-21T08:43:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766f400640 1 -- 172.21.15.28:0/227941626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c0af070 con 0x7f76700cde40 2024-03-21T08:43:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.250+0000 7f766f400640 1 -- 172.21.15.28:0/227941626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0aa9e0 con 0x7f76700cde40 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- 172.21.15.28:0/227941626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76700cde40 msgr2=0x7f7670071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76700cde40 0x7f7670071e60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f765c009870 tx=0x7f765c01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- 172.21.15.28:0/227941626 shutdown_connections 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76700cde40 0x7f7670071e60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 --2- 172.21.15.28:0/227941626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7670071a80 0x7f767006f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- 172.21.15.28:0/227941626 >> 172.21.15.28:0/227941626 conn(0x7f76700674f0 msgr2=0x7f767006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- 172.21.15.28:0/227941626 shutdown_connections 2024-03-21T08:43:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- 172.21.15.28:0/227941626 wait complete. 2024-03-21T08:43:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 Processor -- start 2024-03-21T08:43:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.254+0000 7f7677800640 1 -- start start 2024-03-21T08:43:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 0x7f76700b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 0x7f76700b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670168f20 con 0x7f7670071a80 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670144b20 con 0x7f767015cbb0 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76701690a0 con 0x7f76700cde40 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47584/0 (socket says 172.21.15.28:47584) 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 -- 172.21.15.28:0/2098871389 learned_addr learned my addr 172.21.15.28:0/2098871389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7675800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 0x7f76700b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 -- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 msgr2=0x7f76700b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f766fe00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 0x7f76700b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 0x7f76700b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 -- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 msgr2=0x7f76700b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 0x7f76700b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 -- 172.21.15.28:0/2098871389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700b3a70 con 0x7f7670071a80 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7675800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 0x7f76700b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f766fe00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 0x7f76700b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7674e00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f766400d800 tx=0x7f766400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766400b6c0 con 0x7f7670071a80 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7664019070 con 0x7f7670071a80 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76700b3d50 con 0x7f7670071a80 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766400ba60 con 0x7f7670071a80 2024-03-21T08:43:08.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76700ace20 con 0x7f7670071a80 2024-03-21T08:43:08.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.258+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7670073430 con 0x7f7670071a80 2024-03-21T08:43:08.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.262+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f766400bc00 con 0x7f7670071a80 2024-03-21T08:43:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.262+0000 7f766cc00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 0x7f7650069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.262+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7664086680 con 0x7f7670071a80 2024-03-21T08:43:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.262+0000 7f766fe00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 0x7f7650069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.262+0000 7f766fe00640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 0x7f7650069fe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ae000 tx=0x7f765c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.266+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7664056d90 con 0x7f7670071a80 2024-03-21T08:43:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.446+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f767013b460 con 0x7f7670071a80 2024-03-21T08:43:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.446+0000 7f766cc00640 1 -- 172.21.15.28:0/2098871389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7664056690 con 0x7f7670071a80 2024-03-21T08:43:08.450 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 msgr2=0x7f7650069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 0x7f7650069fe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ae000 tx=0x7f765c008040 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 msgr2=0x7f767013a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f766400d800 tx=0x7f766400dcc0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 shutdown_connections 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f767015cbb0 0x7f76700b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7650067b20 0x7f7650069fe0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f76700cde40 0x7f76700b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 --2- 172.21.15.28:0/2098871389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7670071a80 0x7f767013a6a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 >> 172.21.15.28:0/2098871389 conn(0x7f76700674f0 msgr2=0x7f7670144ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 shutdown_connections 2024-03-21T08:43:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.450+0000 7f7677800640 1 -- 172.21.15.28:0/2098871389 wait complete. 2024-03-21T08:43:08.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 Processor -- start 2024-03-21T08:43:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 -- start start 2024-03-21T08:43:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fccec0b1110 0x7fccec06d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0cc3d0 con 0x7fccec09a3a0 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0cc6b0 con 0x7fccec15dff0 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf4400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec0cc180 con 0x7fccec0b1110 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fccec09a3a0 0x7fccec06cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46658/0 (socket says 172.21.15.28:46658) 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf1000640 1 -- 172.21.15.28:0/2775092681 learned_addr learned my addr 172.21.15.28:0/2775092681 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.614+0000 7fccf2400640 1 --2- 172.21.15.28:0/2775092681 >> v2:172.21.15.151:3300/0 conn(0x7fccec0b1110 0x7fccec06d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459546779 0 0) 0x7fccec0cc3d0 con 0x7fccec09a3a0 2024-03-21T08:43:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd4003530 con 0x7fccec09a3a0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684022115 0 0) 0x7fccec0cc6b0 con 0x7fccec15dff0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec0cc3d0 con 0x7fccec15dff0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1692718954 0 0) 0x7fccd4003530 con 0x7fccec09a3a0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 >> v2:172.21.15.151:3300/0 conn(0x7fccec0b1110 msgr2=0x7fccec06d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 --2- 172.21.15.28:0/2775092681 >> v2:172.21.15.151:3300/0 conn(0x7fccec0b1110 0x7fccec06d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 >> v1:172.21.15.151:6789/0 conn(0x7fccec15dff0 legacy=0x7fccec099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec0cc990 con 0x7fccec09a3a0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fccd8001680 con 0x7fccec09a3a0 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf2400640 1 --2- 172.21.15.28:0/2775092681 >> v2:172.21.15.151:3300/0 conn(0x7fccec0b1110 0x7fccec06d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 >> v1:172.21.15.28:6789/0 conn(0x7fccec09a3a0 legacy=0x7fccec06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 0x7fccd4003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4004810 0x7fccd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4005240 0x7fccd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0cc180 con 0x7fccd4004810 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd400c1a0 con 0x7fccd4005240 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd400c3a0 con 0x7fccd4003b50 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1000640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4004810 0x7fccd4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 0x7fccd4003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf2400640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4005240 0x7fccd400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 -- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4005240 msgr2=0x7fccd400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4005240 0x7fccd400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 -- 172.21.15.28:0/2775092681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4004810 msgr2=0x7fccd4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4004810 0x7fccd4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1a00640 1 -- 172.21.15.28:0/2775092681 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4003530 con 0x7fccd4003b50 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf1000640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4004810 0x7fccd4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.618+0000 7fccf2400640 1 -- 172.21.15.28:0/2775092681 reap_dead start 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf1a00640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 0x7fccd4003f50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcce00018d0 tx=0x7fcce001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0592576982e76a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0001520 con 0x7fccd4003b50 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce00052f0 con 0x7fccd4003b50 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccebe00640 1 -- 172.21.15.28:0/2775092681 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0003840 con 0x7fccd4003b50 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 -- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 msgr2=0x7fccd4003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 0x7fccd4003f50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcce00018d0 tx=0x7fcce001c540 comp rx=0 tx=0).stop 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 -- 172.21.15.28:0/2775092681 shutdown_connections 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 --2- 172.21.15.28:0/2775092681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccd4003b50 0x7fccd4003f50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 -- 172.21.15.28:0/2775092681 >> 172.21.15.28:0/2775092681 conn(0x7fccec0b3620 msgr2=0x7fccec0b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 -- 172.21.15.28:0/2775092681 shutdown_connections 2024-03-21T08:43:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.622+0000 7fccf4400640 1 -- 172.21.15.28:0/2775092681 wait complete. 2024-03-21T08:43:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 Processor -- start 2024-03-21T08:43:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- start start 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4003b50 0x7fccec1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccec0b1f20 0x7fccec0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec06d7a0 con 0x7fccec0b6030 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec06d620 con 0x7fccd4003b50 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec0cd190 con 0x7fccec0b1f20 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47598/0 (socket says 172.21.15.28:47598) 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 -- 172.21.15.28:0/3896505220 learned_addr learned my addr 172.21.15.28:0/3896505220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf1000640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccec0b1f20 0x7fccec0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf1a00640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4003b50 0x7fccec1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 -- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccec0b1f20 msgr2=0x7fccec0b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccec0b1f20 0x7fccec0b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 -- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4003b50 msgr2=0x7fccec1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4003b50 0x7fccec1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 -- 172.21.15.28:0/3896505220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec15e940 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf2400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fccec0b46e0 tx=0x7fcce400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce400fd90 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce40039a0 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4003d90 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec15ec20 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.626+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec15f1e0 con 0x7fccec0b6030 2024-03-21T08:43:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec15f3e0 con 0x7fccec0b6030 2024-03-21T08:43:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcce401f020 con 0x7fccec0b6030 2024-03-21T08:43:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fcce9600640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 0x7fcccc06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcce40865e0 con 0x7fccec0b6030 2024-03-21T08:43:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fccf1a00640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 0x7fcccc06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.630+0000 7fccf1a00640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 0x7fcccc06a010 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcce00ac000 tx=0x7fcce00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.634+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fccec15f3e0 con 0x7fccec0b6030 2024-03-21T08:43:08.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fccec04f4f0 con 0x7fccec0b6030 2024-03-21T08:43:08.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fcce9600640 1 -- 172.21.15.28:0/3896505220 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcce4010040 con 0x7fccec0b6030 2024-03-21T08:43:08.800 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 msgr2=0x7fcccc06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 0x7fcccc06a010 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcce00ac000 tx=0x7fcce00a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 msgr2=0x7fccec0b64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fccec0b46e0 tx=0x7fcce400f580 comp rx=0 tx=0).stop 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 shutdown_connections 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccec0b6030 0x7fccec0b64b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcccc067b50 0x7fcccc06a010 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fccec0b1f20 0x7fccec0b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 --2- 172.21.15.28:0/3896505220 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fccd4003b50 0x7fccec1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 >> 172.21.15.28:0/3896505220 conn(0x7fccec0b3620 msgr2=0x7fccec0706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 shutdown_connections 2024-03-21T08:43:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.798+0000 7fccf4400640 1 -- 172.21.15.28:0/3896505220 wait complete. 2024-03-21T08:43:08.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:43:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.962+0000 7f49c9800640 1 Processor -- start 2024-03-21T08:43:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.962+0000 7f49c9800640 1 -- start start 2024-03-21T08:43:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49c40d1360 0x7f49c40b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c4161a20 con 0x7f49c40ced70 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c9800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c40d2170 con 0x7f49c415e3a0 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c40b3730 con 0x7f49c40d1360 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f49c40ced70 0x7f49c40d1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46662/0 (socket says 172.21.15.28:46662) 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2a00640 1 -- 172.21.15.28:0/3913168568 learned_addr learned my addr 172.21.15.28:0/3913168568 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c3400640 1 --2- 172.21.15.28:0/3913168568 >> v2:172.21.15.28:3300/0 conn(0x7f49c40d1360 0x7f49c40b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887691894 0 0) 0x7f49c4161a20 con 0x7f49c40ced70 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8003530 con 0x7f49c40ced70 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388144129 0 0) 0x7f49c40d2170 con 0x7f49c415e3a0 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c4161a20 con 0x7f49c415e3a0 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c3400640 1 -- 172.21.15.28:0/3913168568 >> v1:172.21.15.151:6789/0 conn(0x7f49c415e3a0 legacy=0x7f49c40d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c3400640 1 -- 172.21.15.28:0/3913168568 >> v1:172.21.15.28:6789/0 conn(0x7f49c40ced70 legacy=0x7f49c40d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c3400640 1 -- 172.21.15.28:0/3913168568 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c415e9e0 con 0x7f49c40d1360 2024-03-21T08:43:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.966+0000 7f49c3400640 1 --2- 172.21.15.28:0/3913168568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40d1360 0x7f49c40b31f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d90 tx=0x7f49bc01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce29997ae1c4ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0b2070 con 0x7f49c40d1360 2024-03-21T08:43:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc0a5910 con 0x7f49c40d1360 2024-03-21T08:43:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2000640 1 -- 172.21.15.28:0/3913168568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a5ce0 con 0x7f49c40d1360 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- 172.21.15.28:0/3913168568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40d1360 msgr2=0x7f49c40b31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 --2- 172.21.15.28:0/3913168568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40d1360 0x7f49c40b31f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d90 tx=0x7f49bc01c940 comp rx=0 tx=0).stop 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- 172.21.15.28:0/3913168568 shutdown_connections 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 --2- 172.21.15.28:0/3913168568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40d1360 0x7f49c40b31f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- 172.21.15.28:0/3913168568 >> 172.21.15.28:0/3913168568 conn(0x7f49c4072aa0 msgr2=0x7f49c4072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- 172.21.15.28:0/3913168568 shutdown_connections 2024-03-21T08:43:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- 172.21.15.28:0/3913168568 wait complete. 2024-03-21T08:43:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 Processor -- start 2024-03-21T08:43:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- start start 2024-03-21T08:43:08.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 0x7f49c40d25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 0x7f49c40d2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c40cf2d0 con 0x7f49c40ced70 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c40cf150 con 0x7f49c40d1360 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c9800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c4157a80 con 0x7f49c415e3a0 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47612/0 (socket says 172.21.15.28:47612) 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.970+0000 7f49c2a00640 1 -- 172.21.15.28:0/1154125654 learned_addr learned my addr 172.21.15.28:0/1154125654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49bbe00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 0x7f49c40d25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c3400640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 0x7f49c40d2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 -- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 msgr2=0x7f49c40d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 0x7f49c40d2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 -- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 msgr2=0x7f49c40d25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 0x7f49c40d25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 -- 172.21.15.28:0/1154125654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c414dab0 con 0x7f49c40ced70 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49bbe00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 0x7f49c40d25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c3400640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 0x7f49c40d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c2a00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c520 tx=0x7f49b400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4017070 con 0x7f49c40ced70 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b4005cf0 con 0x7f49c40ced70 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c414dd90 con 0x7f49c40ced70 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4007270 con 0x7f49c40ced70 2024-03-21T08:43:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c414e350 con 0x7f49c40ced70 2024-03-21T08:43:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c408afc0 con 0x7f49c40ced70 2024-03-21T08:43:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f49b401a070 con 0x7f49c40ced70 2024-03-21T08:43:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 0x7f49a0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.974+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f49b4085550 con 0x7f49c40ced70 2024-03-21T08:43:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.978+0000 7f49bbe00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 0x7f49a0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.978+0000 7f49bbe00640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 0x7f49a0069fe0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49c414ec50 tx=0x7f49ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:08.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:08.978+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f49b4055580 con 0x7f49c40ced70 2024-03-21T08:43:09.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f49c4157860 con 0x7f49c40ced70 2024-03-21T08:43:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49baa00640 1 -- 172.21.15.28:0/1154125654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7f49b4059620 con 0x7f49c40ced70 2024-03-21T08:43:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 msgr2=0x7f49a0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 0x7f49a0069fe0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49c414ec50 tx=0x7f49ac009040 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 msgr2=0x7f49c40b8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c520 tx=0x7f49b400c9e0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 shutdown_connections 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f49a0067b20 0x7f49a0069fe0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f49c415e3a0 0x7f49c40d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f49c40d1360 0x7f49c40d25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 --2- 172.21.15.28:0/1154125654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c40ced70 0x7f49c40b8740 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 >> 172.21.15.28:0/1154125654 conn(0x7f49c4072aa0 msgr2=0x7f49c414eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 shutdown_connections 2024-03-21T08:43:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.146+0000 7f49c9800640 1 -- 172.21.15.28:0/1154125654 wait complete. 2024-03-21T08:43:09.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 Processor -- start 2024-03-21T08:43:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 -- start start 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc068162c90 0x7fc0680b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc068163070 0x7fc068071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06806ea00 con 0x7fc068163500 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680b5000 con 0x7fc068162c90 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06fc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680b4dd0 con 0x7fc068163070 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc068162c90 0x7fc0680b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc068162c90 0x7fc0680b4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47616/0 (socket says 172.21.15.28:47616) 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 -- 172.21.15.28:0/928592871 learned_addr learned my addr 172.21.15.28:0/928592871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc05fe00640 1 --2- 172.21.15.28:0/928592871 >> v2:172.21.15.151:3300/0 conn(0x7fc068163070 0x7fc068071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 -- 172.21.15.28:0/928592871 >> v2:172.21.15.151:3300/0 conn(0x7fc068163070 msgr2=0x7fc068071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 --2- 172.21.15.28:0/928592871 >> v2:172.21.15.151:3300/0 conn(0x7fc068163070 0x7fc068071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 -- 172.21.15.28:0/928592871 >> v1:172.21.15.151:6789/0 conn(0x7fc068163500 legacy=0x7fc068165d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.318+0000 7fc06d200640 1 -- 172.21.15.28:0/928592871 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06806dc40 con 0x7fc068162c90 2024-03-21T08:43:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06d200640 1 --2- 172.21.15.28:0/928592871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc0680b4890 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc06806fd80 tx=0x7fc05801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa62860dab16645d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc067e00640 1 -- 172.21.15.28:0/928592871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580b0070 con 0x7fc068162c90 2024-03-21T08:43:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc067e00640 1 -- 172.21.15.28:0/928592871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0580ab600 con 0x7fc068162c90 2024-03-21T08:43:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc067e00640 1 -- 172.21.15.28:0/928592871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580ab9d0 con 0x7fc068162c90 2024-03-21T08:43:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- 172.21.15.28:0/928592871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 msgr2=0x7fc0680b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- 172.21.15.28:0/928592871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc0680b4890 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc06806fd80 tx=0x7fc05801c520 comp rx=0 tx=0).stop 2024-03-21T08:43:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- 172.21.15.28:0/928592871 shutdown_connections 2024-03-21T08:43:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- 172.21.15.28:0/928592871 >> v2:172.21.15.151:3300/0 conn(0x7fc068163070 0x7fc068071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- 172.21.15.28:0/928592871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc0680b4890 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- 172.21.15.28:0/928592871 >> 172.21.15.28:0/928592871 conn(0x7fc0680674f0 msgr2=0x7fc06806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- 172.21.15.28:0/928592871 shutdown_connections 2024-03-21T08:43:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- 172.21.15.28:0/928592871 wait complete. 2024-03-21T08:43:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 Processor -- start 2024-03-21T08:43:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- start start 2024-03-21T08:43:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc068163070 0x7fc06814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06814fcc0 0x7fc06814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06816f0e0 con 0x7fc068162c90 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680b3ae0 con 0x7fc06814fcc0 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.326+0000 7fc06fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06816f260 con 0x7fc068163070 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47632/0 (socket says 172.21.15.28:47632) 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 -- 172.21.15.28:0/4131571086 learned_addr learned my addr 172.21.15.28:0/4131571086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06dc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06814fcc0 0x7fc06814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc05fe00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc068163070 0x7fc06814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 -- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc068163070 msgr2=0x7fc06814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc068163070 0x7fc06814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 -- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06814fcc0 msgr2=0x7fc06814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06814fcc0 0x7fc06814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 -- 172.21.15.28:0/4131571086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06815f200 con 0x7fc068162c90 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06d200640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc05801ca00 tx=0x7fc0580192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580b0040 con 0x7fc068162c90 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0580b9070 con 0x7fc068162c90 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580b4660 con 0x7fc068162c90 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06814dba0 con 0x7fc068162c90 2024-03-21T08:43:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06814e160 con 0x7fc068162c90 2024-03-21T08:43:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0681500d0 con 0x7fc068162c90 2024-03-21T08:43:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.330+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc0580b4800 con 0x7fc068162c90 2024-03-21T08:43:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.334+0000 7fc065600640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 0x7fc044069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.334+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc05811fe90 con 0x7fc068162c90 2024-03-21T08:43:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.334+0000 7fc05fe00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 0x7fc044069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.334+0000 7fc05fe00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 0x7fc044069fe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc068159700 tx=0x7fc050008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.338+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc0681500d0 con 0x7fc068162c90 2024-03-21T08:43:09.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.518+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc068073430 con 0x7fc068162c90 2024-03-21T08:43:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.518+0000 7fc065600640 1 -- 172.21.15.28:0/4131571086 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v42) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc058016110 con 0x7fc068162c90 2024-03-21T08:43:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 msgr2=0x7fc044069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 0x7fc044069fe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc068159700 tx=0x7fc050008040 comp rx=0 tx=0).stop 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 msgr2=0x7fc06814c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc05801ca00 tx=0x7fc0580192c0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 shutdown_connections 2024-03-21T08:43:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc06814fcc0 0x7fc06814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc044067b20 0x7fc044069fe0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc068163070 0x7fc06814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 --2- 172.21.15.28:0/4131571086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc068162c90 0x7fc06814c760 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 >> 172.21.15.28:0/4131571086 conn(0x7fc0680674f0 msgr2=0x7fc068152b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 shutdown_connections 2024-03-21T08:43:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:09.522+0000 7fc06fc00640 1 -- 172.21.15.28:0/4131571086 wait complete. 2024-03-21T08:43:09.543 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:43:09.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:09.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:09.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[38546]: fuse finished with error 0 and tester_r 0 2024-03-21T08:43:09.566 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:15.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:43:15.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:15.668 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:43:15.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-21T08:43:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 Processor -- start 2024-03-21T08:43:15.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 -- start start 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 0x7fd2ec06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0b3010 con 0x7fd2ec06cbe0 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec06d310 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2f0e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec06d5f0 con 0x7fd2ec0d0190 2024-03-21T08:43:15.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42372/0 (socket says 172.21.15.28:42372) 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 -- 172.21.15.28:0/3025928026 learned_addr learned my addr 172.21.15.28:0/3025928026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9400640 1 --2- 172.21.15.28:0/3025928026 >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 0x7fd2ec06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 -- 172.21.15.28:0/3025928026 >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 msgr2=0x7fd2ec06c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 --2- 172.21.15.28:0/3025928026 >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 0x7fd2ec06c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 -- 172.21.15.28:0/3025928026 >> v1:172.21.15.28:6789/0 conn(0x7fd2ec06cbe0 legacy=0x7fd2ec161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9e00640 1 -- 172.21.15.28:0/3025928026 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec06d8d0 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e9400640 1 --2- 172.21.15.28:0/3025928026 >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 0x7fd2ec06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:15.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.866+0000 7fd2e8a00640 1 -- 172.21.15.28:0/3025928026 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487104629 0 0) 0x7fd2ec0b3010 con 0x7fd2ec06cbe0 2024-03-21T08:43:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2e9e00640 1 --2- 172.21.15.28:0/3025928026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec067870 tx=0x7fd2dc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c94c7bdc5390a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2e8a00640 1 -- 172.21.15.28:0/3025928026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0b0070 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2e8a00640 1 -- 172.21.15.28:0/3025928026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc0ab600 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2e8a00640 1 -- 172.21.15.28:0/3025928026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0ab9d0 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3025928026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec15e1e0 msgr2=0x7fd2ec0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.870+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3025928026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec067870 tx=0x7fd2dc01c520 comp rx=0 tx=0).stop 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3025928026 shutdown_connections 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3025928026 >> v2:172.21.15.151:3300/0 conn(0x7fd2ec0d0190 0x7fd2ec06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3025928026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec15e1e0 0x7fd2ec0cfc50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3025928026 >> 172.21.15.28:0/3025928026 conn(0x7fd2ec0d1ab0 msgr2=0x7fd2ec0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3025928026 shutdown_connections 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3025928026 wait complete. 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 Processor -- start 2024-03-21T08:43:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- start start 2024-03-21T08:43:15.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2ec06cbe0 0x7fd2ec0b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2ec15e1e0 0x7fd2ec150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec16b2d0 con 0x7fd2ec0d0190 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec16b150 con 0x7fd2ec06cbe0 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2f0e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec16b450 con 0x7fd2ec15e1e0 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42386/0 (socket says 172.21.15.28:42386) 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 -- 172.21.15.28:0/3202067669 learned_addr learned my addr 172.21.15.28:0/3202067669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2ec06cbe0 0x7fd2ec0b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2ea800640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2ec15e1e0 0x7fd2ec150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 -- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2ec15e1e0 msgr2=0x7fd2ec150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2ec15e1e0 0x7fd2ec150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 -- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2ec06cbe0 msgr2=0x7fd2ec0b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2ec06cbe0 0x7fd2ec0b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 -- 172.21.15.28:0/3202067669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec152010 con 0x7fd2ec0d0190 2024-03-21T08:43:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.874+0000 7fd2e9400640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800d830 tx=0x7fd2d800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800b6c0 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8019070 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec151770 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800baa0 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec151d30 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec04f4f0 con 0x7fd2ec0d0190 2024-03-21T08:43:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd2d801e460 con 0x7fd2ec0d0190 2024-03-21T08:43:15.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 0x7fd2cc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:15.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.878+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd2d8086780 con 0x7fd2ec0d0190 2024-03-21T08:43:15.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.882+0000 7fd2e9e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 0x7fd2cc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.882+0000 7fd2e9e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 0x7fd2cc069fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0ad000 tx=0x7fd2dc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:15.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:15.882+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd2d80567b0 con 0x7fd2ec0d0190 2024-03-21T08:43:16.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.106+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd2ec0d2830 con 0x7fd2ec0d0190 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.106+0000 7fd2e2000640 1 -- 172.21.15.28:0/3202067669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+141 (secure 0 0 0) 0x7fd2d805a850 con 0x7fd2ec0d0190 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-21T08:43:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:43:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 msgr2=0x7fd2cc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 0x7fd2cc069fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc0ad000 tx=0x7fd2dc0a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 msgr2=0x7fd2ec150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800d830 tx=0x7fd2d800dcf0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 shutdown_connections 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd2cc067b20 0x7fd2cc069fe0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd2ec15e1e0 0x7fd2ec150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2ec0d0190 0x7fd2ec150d20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 --2- 172.21.15.28:0/3202067669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd2ec06cbe0 0x7fd2ec0b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 >> 172.21.15.28:0/3202067669 conn(0x7fd2ec0d1ab0 msgr2=0x7fd2ec0ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 shutdown_connections 2024-03-21T08:43:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.110+0000 7fd2f0e00640 1 -- 172.21.15.28:0/3202067669 wait complete. 2024-03-21T08:43:16.133 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-21T08:43:16.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:43:16.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:43:16.273 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-21T08:43:16.273 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:16.274 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:16.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:43:16.283 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:43:16.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:16.283 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:43:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-21T08:43:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:43:16.335 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:43:16.335 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:16.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:43:16.344 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:43:16.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:16.345 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-21T08:43:16.394 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-21T08:43:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:43:16.395 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:16.395 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:16.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:43:16.404 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:43:16.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:16.404 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-21T08:43:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-21T08:43:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:43:16.455 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:43:16.455 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:16.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:43:16.464 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:43:16.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:16.465 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-21T08:43:16.514 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-21T08:43:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:43:16.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.654+0000 7ff85be00640 1 Processor -- start 2024-03-21T08:43:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.654+0000 7ff85be00640 1 -- start start 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85be00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff85c0b0ae0 0x7ff85c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0d0380 con 0x7ff85c06c490 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85be00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0d0660 con 0x7ff85c09a140 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85be00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85c0d0150 con 0x7ff85c0b0ae0 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff85c06c490 0x7ff85c099a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43172/0 (socket says 172.21.15.28:43172) 2024-03-21T08:43:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 -- 172.21.15.28:0/3308943322 learned_addr learned my addr 172.21.15.28:0/3308943322 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85aa00640 1 --2- 172.21.15.28:0/3308943322 >> v2:172.21.15.151:3300/0 conn(0x7ff85c0b0ae0 0x7ff85c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134911875 0 0) 0x7ff85c0d0380 con 0x7ff85c06c490 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840003530 con 0x7ff85c06c490 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126506571 0 0) 0x7ff840003530 con 0x7ff85c06c490 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 >> v2:172.21.15.151:3300/0 conn(0x7ff85c0b0ae0 msgr2=0x7ff85c0cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 --2- 172.21.15.28:0/3308943322 >> v2:172.21.15.151:3300/0 conn(0x7ff85c0b0ae0 0x7ff85c0cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 >> v1:172.21.15.151:6789/0 conn(0x7ff85c09a140 legacy=0x7ff85c15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0d0940 con 0x7ff85c06c490 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7ff844001680 con 0x7ff85c06c490 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 >> v1:172.21.15.28:6789/0 conn(0x7ff85c06c490 legacy=0x7ff85c099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8400039c0 0x7ff840003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 0x7ff840004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff840005090 0x7ff84000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84000bfc0 con 0x7ff8400043c0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84000c170 con 0x7ff8400039c0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84000c370 con 0x7ff840005090 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 0x7ff840004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85aa00640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8400039c0 0x7ff840003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85b400640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff840005090 0x7ff84000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 -- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff840005090 msgr2=0x7ff84000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff840005090 0x7ff84000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 -- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8400039c0 msgr2=0x7ff840003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff8400039c0 0x7ff840003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff852000640 1 -- 172.21.15.28:0/3308943322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840003530 con 0x7ff8400043c0 2024-03-21T08:43:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.658+0000 7ff85b400640 1 -- 172.21.15.28:0/3308943322 reap_dead start 2024-03-21T08:43:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff852000640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 0x7ff840004890 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c290 tx=0x7ff84401cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4f0a1d8108607e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844007d60 con 0x7ff8400043c0 2024-03-21T08:43:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8440a9070 con 0x7ff8400043c0 2024-03-21T08:43:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85a000640 1 -- 172.21.15.28:0/3308943322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844005d80 con 0x7ff8400043c0 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 msgr2=0x7ff840004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 0x7ff840004890 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c290 tx=0x7ff84401cd00 comp rx=0 tx=0).stop 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- 172.21.15.28:0/3308943322 shutdown_connections 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 --2- 172.21.15.28:0/3308943322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8400043c0 0x7ff840004890 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- 172.21.15.28:0/3308943322 >> 172.21.15.28:0/3308943322 conn(0x7ff85c0b3390 msgr2=0x7ff85c0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- 172.21.15.28:0/3308943322 shutdown_connections 2024-03-21T08:43:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- 172.21.15.28:0/3308943322 wait complete. 2024-03-21T08:43:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 Processor -- start 2024-03-21T08:43:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.662+0000 7ff85be00640 1 -- start start 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff8400043c0 0x7ff85c1fa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 0x7ff85c0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c06d860 con 0x7ff85c0cc380 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0d0f90 con 0x7ff85c0ca630 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c06d9e0 con 0x7ff8400043c0 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff852000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 0x7ff85c0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42392/0 (socket says 172.21.15.28:42392) 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 -- 172.21.15.28:0/1359904838 learned_addr learned my addr 172.21.15.28:0/1359904838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff852000640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 0x7ff85c0cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:48570/0 (socket says 172.21.15.28:48570) 2024-03-21T08:43:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85aa00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff8400043c0 0x7ff85c1fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 -- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff8400043c0 msgr2=0x7ff85c1fa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff8400043c0 0x7ff85c1fa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 -- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 msgr2=0x7ff85c0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 0x7ff85c0cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 -- 172.21.15.28:0/1359904838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0cada0 con 0x7ff85c0cc380 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85b400640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff85c072fc0 tx=0x7ff85400eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854006d70 con 0x7ff85c0cc380 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854003980 con 0x7ff85c0cc380 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854003d70 con 0x7ff85c0cc380 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c0cb080 con 0x7ff85c0cc380 2024-03-21T08:43:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c0cb640 con 0x7ff85c0cc380 2024-03-21T08:43:16.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.666+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c075270 con 0x7ff85c0cc380 2024-03-21T08:43:16.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.670+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff854005070 con 0x7ff85c0cc380 2024-03-21T08:43:16.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.670+0000 7ff853e00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 0x7ff83806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:16.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.670+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff854085880 con 0x7ff85c0cc380 2024-03-21T08:43:16.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.670+0000 7ff85aa00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 0x7ff83806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.670+0000 7ff85aa00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 0x7ff83806a010 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009c60 tx=0x7ff84c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.674+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff8540558b0 con 0x7ff85c0cc380 2024-03-21T08:43:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff85c0cdab0 con 0x7ff85c0cc380 2024-03-21T08:43:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff853e00640 1 -- 172.21.15.28:0/1359904838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4982 (secure 0 0 0) 0x7ff854059950 con 0x7ff85c0cc380 2024-03-21T08:43:16.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:16.877 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:44.201277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:43:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 msgr2=0x7ff83806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 0x7ff83806a010 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009c60 tx=0x7ff84c009190 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 msgr2=0x7ff85c1f78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff85c072fc0 tx=0x7ff85400eab0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 shutdown_connections 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff85c0cc380 0x7ff85c1f78a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff85c0ca630 0x7ff85c0cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff838067b50 0x7ff83806a010 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 --2- 172.21.15.28:0/1359904838 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff8400043c0 0x7ff85c1fa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 >> 172.21.15.28:0/1359904838 conn(0x7ff85c0b3390 msgr2=0x7ff85c06cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 shutdown_connections 2024-03-21T08:43:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:16.874+0000 7ff85be00640 1 -- 172.21.15.28:0/1359904838 wait complete. 2024-03-21T08:43:16.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:17.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 Processor -- start 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 -- start start 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f391c099b80 0x7f391c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 0x7f391c06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f391c0cfac0 con 0x7f391c0d0880 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391c0cfda0 con 0x7f391c099b80 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3923000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391c0d0080 con 0x7f391c0b0b40 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f391c099b80 0x7f391c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f391c099b80 0x7f391c15f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42402/0 (socket says 172.21.15.28:42402) 2024-03-21T08:43:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 -- 172.21.15.28:0/2850521687 learned_addr learned my addr 172.21.15.28:0/2850521687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3921000640 1 --2- 172.21.15.28:0/2850521687 >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 0x7f391c06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 -- 172.21.15.28:0/2850521687 >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 msgr2=0x7f391c06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 --2- 172.21.15.28:0/2850521687 >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 0x7f391c06c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 -- 172.21.15.28:0/2850521687 >> v1:172.21.15.28:6789/0 conn(0x7f391c0d0880 legacy=0x7f391c099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f391be00640 1 -- 172.21.15.28:0/2850521687 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0d0360 con 0x7f391c099b80 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.046+0000 7f3921000640 1 --2- 172.21.15.28:0/2850521687 >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 0x7f391c06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f391be00640 1 --2- 172.21.15.28:0/2850521687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c099b80 0x7f391c15f220 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f391c0737d0 tx=0x7f390c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5b9061c35a400b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f391b400640 1 -- 172.21.15.28:0/2850521687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c0ab640 con 0x7f391c099b80 2024-03-21T08:43:17.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f391b400640 1 -- 172.21.15.28:0/2850521687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c0b0070 con 0x7f391c099b80 2024-03-21T08:43:17.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f391b400640 1 -- 172.21.15.28:0/2850521687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c0aba00 con 0x7f391c099b80 2024-03-21T08:43:17.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 -- 172.21.15.28:0/2850521687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c099b80 msgr2=0x7f391c15f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 --2- 172.21.15.28:0/2850521687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c099b80 0x7f391c15f220 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f391c0737d0 tx=0x7f390c01c520 comp rx=0 tx=0).stop 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 -- 172.21.15.28:0/2850521687 shutdown_connections 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 --2- 172.21.15.28:0/2850521687 >> v2:172.21.15.151:3300/0 conn(0x7f391c0b0b40 0x7f391c06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 --2- 172.21.15.28:0/2850521687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c099b80 0x7f391c15f220 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 -- 172.21.15.28:0/2850521687 >> 172.21.15.28:0/2850521687 conn(0x7f391c0b3510 msgr2=0x7f391c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 -- 172.21.15.28:0/2850521687 shutdown_connections 2024-03-21T08:43:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.050+0000 7f3923000640 1 -- 172.21.15.28:0/2850521687 wait complete. 2024-03-21T08:43:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 Processor -- start 2024-03-21T08:43:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- start start 2024-03-21T08:43:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f391c099b80 0x7f391c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 0x7f391c14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c06d860 con 0x7f391c0d0880 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c0b2d40 con 0x7f391c0b0b40 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391c16b050 con 0x7f391c099b80 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42414/0 (socket says 172.21.15.28:42414) 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 -- 172.21.15.28:0/2996780843 learned_addr learned my addr 172.21.15.28:0/2996780843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 -- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f391c099b80 msgr2=0x7f391c149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f391c099b80 0x7f391c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 -- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 msgr2=0x7f391c14e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3913e00640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 0x7f391c14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 0x7f391c14e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 -- 172.21.15.28:0/2996780843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0b5400 con 0x7f391c0d0880 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3913e00640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 0x7f391c14e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3921000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f391400cd50 tx=0x7f391400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391400f680 con 0x7f391c0d0880 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391400f840 con 0x7f391c0d0880 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391400fc30 con 0x7f391c0d0880 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c158b10 con 0x7f391c0d0880 2024-03-21T08:43:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.054+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c1590d0 con 0x7f391c0d0880 2024-03-21T08:43:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.058+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c04f410 con 0x7f391c0d0880 2024-03-21T08:43:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.058+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f39140213f0 con 0x7f391c0d0880 2024-03-21T08:43:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.058+0000 7f3918c00640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 0x7f38f8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.058+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3914086fb0 con 0x7f391c0d0880 2024-03-21T08:43:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.062+0000 7f391be00640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 0x7f38f8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.062+0000 7f391be00640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 0x7f38f8069fe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f390c0af000 tx=0x7f390c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.066+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3914014030 con 0x7f391c0d0880 2024-03-21T08:43:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.230+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f391c0b41b0 con 0x7f391c0d0880 2024-03-21T08:43:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.230+0000 7f3918c00640 1 -- 172.21.15.28:0/2996780843 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f39140576b0 con 0x7f391c0d0880 2024-03-21T08:43:17.233 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 msgr2=0x7f38f8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 0x7f38f8069fe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f390c0af000 tx=0x7f390c004740 comp rx=0 tx=0).stop 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 msgr2=0x7f391c0b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f391400cd50 tx=0x7f391400a650 comp rx=0 tx=0).stop 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 shutdown_connections 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391c0d0880 0x7f391c0b4820 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f391c0b0b40 0x7f391c14e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f38f8067b20 0x7f38f8069fe0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 --2- 172.21.15.28:0/2996780843 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f391c099b80 0x7f391c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 >> 172.21.15.28:0/2996780843 conn(0x7f391c0b3510 msgr2=0x7f391c0ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 shutdown_connections 2024-03-21T08:43:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.234+0000 7f3923000640 1 -- 172.21.15.28:0/2996780843 wait complete. 2024-03-21T08:43:17.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:42:27.937645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:42:29.064158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-21T08:42:43.385115+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:17.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.394+0000 7f316fa00640 1 Processor -- start 2024-03-21T08:43:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316fa00640 1 -- start start 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31680cc640 0x7f31680cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316fa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680ccbd0 con 0x7f3168099c90 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316fa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680cceb0 con 0x7f316815bdc0 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31680cd7e0 con 0x7f31680cc640 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31680cc640 0x7f31680cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31680cc640 0x7f31680cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42428/0 (socket says 172.21.15.28:42428) 2024-03-21T08:43:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 -- 172.21.15.28:0/3535793923 learned_addr learned my addr 172.21.15.28:0/3535793923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f316da00640 1 --1- 172.21.15.28:0/3535793923 >> v1:172.21.15.28:6789/0 conn(0x7f3168099c90 0x7f31680b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43176/0 (socket says 172.21.15.28:43176) 2024-03-21T08:43:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 -- 172.21.15.28:0/3535793923 >> v1:172.21.15.151:6789/0 conn(0x7f316815bdc0 legacy=0x7f31680d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 -- 172.21.15.28:0/3535793923 >> v1:172.21.15.28:6789/0 conn(0x7f3168099c90 legacy=0x7f31680b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167e00640 1 -- 172.21.15.28:0/3535793923 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31680cd190 con 0x7f31680cc640 2024-03-21T08:43:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.398+0000 7f3167400640 1 -- 172.21.15.28:0/3535793923 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967697858 0 0) 0x7f31680ccbd0 con 0x7f3168099c90 2024-03-21T08:43:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.402+0000 7f3167e00640 1 --2- 172.21.15.28:0/3535793923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31680cc640 0x7f31680cdc70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be4b69cf9917b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.402+0000 7f3167400640 1 -- 172.21.15.28:0/3535793923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540aa600 con 0x7f31680cc640 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.402+0000 7f3167400640 1 -- 172.21.15.28:0/3535793923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31540af070 con 0x7f31680cc640 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.402+0000 7f3167400640 1 -- 172.21.15.28:0/3535793923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540aa9c0 con 0x7f31680cc640 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- 172.21.15.28:0/3535793923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31680cc640 msgr2=0x7f31680cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 --2- 172.21.15.28:0/3535793923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31680cc640 0x7f31680cdc70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- 172.21.15.28:0/3535793923 shutdown_connections 2024-03-21T08:43:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 --2- 172.21.15.28:0/3535793923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31680cc640 0x7f31680cdc70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- 172.21.15.28:0/3535793923 >> 172.21.15.28:0/3535793923 conn(0x7f31680b3620 msgr2=0x7f31680b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- 172.21.15.28:0/3535793923 shutdown_connections 2024-03-21T08:43:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- 172.21.15.28:0/3535793923 wait complete. 2024-03-21T08:43:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 Processor -- start 2024-03-21T08:43:17.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- start start 2024-03-21T08:43:17.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 0x7f316814d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680cc640 0x7f31680cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680b1e50 con 0x7f316815bdc0 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316809a4f0 con 0x7f31680cc640 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316fa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680b1fd0 con 0x7f3168099c90 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42432/0 (socket says 172.21.15.28:42432) 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 -- 172.21.15.28:0/1772062699 learned_addr learned my addr 172.21.15.28:0/1772062699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f3167e00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680cc640 0x7f31680cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316d000640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 0x7f316814d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 -- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 msgr2=0x7f316814d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 0x7f316814d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 -- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680cc640 msgr2=0x7f31680cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680cc640 0x7f31680cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 -- 172.21.15.28:0/1772062699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31681653f0 con 0x7f316815bdc0 2024-03-21T08:43:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316d000640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 0x7f316814d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.406+0000 7f316da00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f316000c700 tx=0x7f316000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160018070 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316000ce10 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316000a860 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31681656d0 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31680c97a0 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316809faf0 con 0x7f316815bdc0 2024-03-21T08:43:17.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f316000aa00 con 0x7f316815bdc0 2024-03-21T08:43:17.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 0x7f3148069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3160085820 con 0x7f316815bdc0 2024-03-21T08:43:17.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.410+0000 7f316d000640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 0x7f3148069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.414+0000 7f316d000640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 0x7f3148069fe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f31681579e0 tx=0x7f315c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.414+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3160055850 con 0x7f316815bdc0 2024-03-21T08:43:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.578+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31680d0d90 con 0x7f316815bdc0 2024-03-21T08:43:17.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.578+0000 7f3164c00640 1 -- 172.21.15.28:0/1772062699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31600598f0 con 0x7f316815bdc0 2024-03-21T08:43:17.582 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 msgr2=0x7f3148069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 0x7f3148069fe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f31681579e0 tx=0x7f315c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 msgr2=0x7f31680cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f316000c700 tx=0x7f316000cbc0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 shutdown_connections 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f316815bdc0 0x7f31680cb280 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f31680cc640 0x7f31680cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3148067b20 0x7f3148069fe0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 --2- 172.21.15.28:0/1772062699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f3168099c90 0x7f316814d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 >> 172.21.15.28:0/1772062699 conn(0x7f31680b3620 msgr2=0x7f31680712b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 shutdown_connections 2024-03-21T08:43:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.582+0000 7f316fa00640 1 -- 172.21.15.28:0/1772062699 wait complete. 2024-03-21T08:43:17.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:17.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.746+0000 7fc9a3a00640 1 Processor -- start 2024-03-21T08:43:17.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.746+0000 7fc9a3a00640 1 -- start start 2024-03-21T08:43:17.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 0x7fc99c06f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c070970 0x7fc99c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc99c0af9b0 0x7fc99c06b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c0cd3f0 con 0x7fc99c0af9b0 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c0ce180 con 0x7fc99c070590 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a3a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c06ecb0 con 0x7fc99c070970 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc99c0af9b0 0x7fc99c06b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc99c0af9b0 0x7fc99c06b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42438/0 (socket says 172.21.15.28:42438) 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 -- 172.21.15.28:0/1214420026 learned_addr learned my addr 172.21.15.28:0/1214420026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1000640 1 --2- 172.21.15.28:0/1214420026 >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 0x7fc99c06f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc99be00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c070970 0x7fc99c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 -- 172.21.15.28:0/1214420026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c070970 msgr2=0x7fc99c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c070970 0x7fc99c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 -- 172.21.15.28:0/1214420026 >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 msgr2=0x7fc99c06f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 --2- 172.21.15.28:0/1214420026 >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 0x7fc99c06f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1a00640 1 -- 172.21.15.28:0/1214420026 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0cdbc0 con 0x7fc99c0af9b0 2024-03-21T08:43:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.750+0000 7fc9a1000640 1 --2- 172.21.15.28:0/1214420026 >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 0x7fc99c06f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a1a00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c0af9b0 0x7fc99c06b6b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc99c072ca0 tx=0x7fc99401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ac73ca4d1a70f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc99b400640 1 -- 172.21.15.28:0/1214420026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940a57d0 con 0x7fc99c0af9b0 2024-03-21T08:43:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc99b400640 1 -- 172.21.15.28:0/1214420026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9940a5970 con 0x7fc99c0af9b0 2024-03-21T08:43:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc99b400640 1 -- 172.21.15.28:0/1214420026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940a5d40 con 0x7fc99c0af9b0 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1214420026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c0af9b0 msgr2=0x7fc99c06b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c0af9b0 0x7fc99c06b6b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc99c072ca0 tx=0x7fc99401c930 comp rx=0 tx=0).stop 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1214420026 shutdown_connections 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c0af9b0 0x7fc99c06b6b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1214420026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c070970 0x7fc99c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1214420026 >> v2:172.21.15.151:3300/0 conn(0x7fc99c070590 0x7fc99c06f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1214420026 >> 172.21.15.28:0/1214420026 conn(0x7fc99c099a30 msgr2=0x7fc99c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1214420026 shutdown_connections 2024-03-21T08:43:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1214420026 wait complete. 2024-03-21T08:43:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 Processor -- start 2024-03-21T08:43:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- start start 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc99c070590 0x7fc99c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c0af9b0 0x7fc99c1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0cb950 con 0x7fc99c070970 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0cb7d0 con 0x7fc99c070590 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc9a3a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c15d5f0 con 0x7fc99c0af9b0 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.754+0000 7fc99be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42448/0 (socket says 172.21.15.28:42448) 2024-03-21T08:43:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 -- 172.21.15.28:0/1533672876 learned_addr learned my addr 172.21.15.28:0/1533672876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc9a1000640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc99c070590 0x7fc99c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc9a1a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c0af9b0 0x7fc99c1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 -- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c0af9b0 msgr2=0x7fc99c1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c0af9b0 0x7fc99c1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 -- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc99c070590 msgr2=0x7fc99c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc99c070590 0x7fc99c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 -- 172.21.15.28:0/1533672876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c15ebc0 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc99be00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc98800d800 tx=0x7fc98800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98800b6c0 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc988019070 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c0b21b0 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98800ba60 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c15f140 con 0x7fc99c070970 2024-03-21T08:43:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.758+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c04f4f0 con 0x7fc99c070970 2024-03-21T08:43:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.762+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc98800bc00 con 0x7fc99c070970 2024-03-21T08:43:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.762+0000 7fc998c00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 0x7fc97c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.762+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc9880864a0 con 0x7fc99c070970 2024-03-21T08:43:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.762+0000 7fc9a1000640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 0x7fc97c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.762+0000 7fc9a1000640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 0x7fc97c06a010 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc990009950 tx=0x7fc990008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.766+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc988056550 con 0x7fc99c070970 2024-03-21T08:43:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.926+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc99c0cd5d0 con 0x7fc99c070970 2024-03-21T08:43:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc998c00640 1 -- 172.21.15.28:0/1533672876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc98805a5f0 con 0x7fc99c070970 2024-03-21T08:43:17.931 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 msgr2=0x7fc97c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 0x7fc97c06a010 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc990009950 tx=0x7fc990008040 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 msgr2=0x7fc99c0b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc98800d800 tx=0x7fc98800dcc0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 shutdown_connections 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc97c067b50 0x7fc97c06a010 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc99c0af9b0 0x7fc99c1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc99c070970 0x7fc99c0b1a40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 --2- 172.21.15.28:0/1533672876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc99c070590 0x7fc99c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 >> 172.21.15.28:0/1533672876 conn(0x7fc99c099a30 msgr2=0x7fc99c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 shutdown_connections 2024-03-21T08:43:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:17.930+0000 7fc9a3a00640 1 -- 172.21.15.28:0/1533672876 wait complete. 2024-03-21T08:43:17.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 Processor -- start 2024-03-21T08:43:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 -- start start 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0d406b880 0x7fd0d40ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d40cfbb0 0x7fd0d40cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40cebc0 con 0x7fd0d4152b60 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d40ceea0 con 0x7fd0d406b880 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d9400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40cf180 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0c9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0d406b880 0x7fd0d40ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d2800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fd0d4152b60 0x7fd0d406b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:47334/0 (socket says 172.21.15.28:47334) 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d2800640 1 -- 172.21.15.28:0/879306310 learned_addr learned my addr 172.21.15.28:0/879306310 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d3200640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d40cfbb0 0x7fd0d40cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0c9e00640 1 -- 172.21.15.28:0/879306310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d40cfbb0 msgr2=0x7fd0d40cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0c9e00640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d40cfbb0 0x7fd0d40cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0d1e00640 1 -- 172.21.15.28:0/879306310 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265593640 0 0) 0x7fd0d40cebc0 con 0x7fd0d4152b60 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0c9e00640 1 -- 172.21.15.28:0/879306310 >> v1:172.21.15.151:6789/0 conn(0x7fd0d4152b60 legacy=0x7fd0d406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.094+0000 7fd0c9e00640 1 -- 172.21.15.28:0/879306310 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40cf460 con 0x7fd0d406b880 2024-03-21T08:43:18.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.098+0000 7fd0c9e00640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d406b880 0x7fd0d40ce520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0097c0 tx=0x7fd0bc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5b699fef7c359f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d1e00640 1 -- 172.21.15.28:0/879306310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0b0070 con 0x7fd0d406b880 2024-03-21T08:43:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d1e00640 1 -- 172.21.15.28:0/879306310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc0ab600 con 0x7fd0d406b880 2024-03-21T08:43:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d1e00640 1 -- 172.21.15.28:0/879306310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0ab9d0 con 0x7fd0d406b880 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- 172.21.15.28:0/879306310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d406b880 msgr2=0x7fd0d40ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d406b880 0x7fd0d40ce520 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0097c0 tx=0x7fd0bc01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- 172.21.15.28:0/879306310 shutdown_connections 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d40cfbb0 0x7fd0d40cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- 172.21.15.28:0/879306310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d406b880 0x7fd0d40ce520 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- 172.21.15.28:0/879306310 >> 172.21.15.28:0/879306310 conn(0x7fd0d4099a30 msgr2=0x7fd0d4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- 172.21.15.28:0/879306310 shutdown_connections 2024-03-21T08:43:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- 172.21.15.28:0/879306310 wait complete. 2024-03-21T08:43:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 Processor -- start 2024-03-21T08:43:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- start start 2024-03-21T08:43:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d406b880 0x7fd0d41519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d4152b60 0x7fd0d41657b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d415c9c0 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d415c840 con 0x7fd0d406b880 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d9400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d415cb40 con 0x7fd0d4152b60 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42472/0 (socket says 172.21.15.28:42472) 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0c9e00640 1 -- 172.21.15.28:0/4075170547 learned_addr learned my addr 172.21.15.28:0/4075170547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d2800640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d406b880 0x7fd0d41519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.102+0000 7fd0d3200640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d4152b60 0x7fd0d41657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 -- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d4152b60 msgr2=0x7fd0d41657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d4152b60 0x7fd0d41657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 -- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d406b880 msgr2=0x7fd0d41519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d406b880 0x7fd0d41519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 -- 172.21.15.28:0/4075170547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4158880 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0c9e00640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0097c0 tx=0x7fd0bc01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0b0040 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc004030 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0043e0 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4158b60 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4159120 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d409ea60 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.106+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd0bc003620 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.110+0000 7fd0cb400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 0x7fd0b0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.110+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd0bc121420 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.110+0000 7fd0d2800640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 0x7fd0b0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.110+0000 7fd0d2800640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 0x7fd0b0069fe0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4000f70 tx=0x7fd0c4007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.114+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd0bc016090 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0d406da80 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0cb400640 1 -- 172.21.15.28:0/4075170547 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd0bc0f1560 con 0x7fd0d40cfbb0 2024-03-21T08:43:18.277 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 msgr2=0x7fd0b0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 0x7fd0b0069fe0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4000f70 tx=0x7fd0c4007df0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 msgr2=0x7fd0d4165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0097c0 tx=0x7fd0bc01ace0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 shutdown_connections 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd0b0067b20 0x7fd0b0069fe0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd0d4152b60 0x7fd0d41657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d40cfbb0 0x7fd0d4165270 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 --2- 172.21.15.28:0/4075170547 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd0d406b880 0x7fd0d41519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.274+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 >> 172.21.15.28:0/4075170547 conn(0x7fd0d4099a30 msgr2=0x7fd0d4159ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.278+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 shutdown_connections 2024-03-21T08:43:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.278+0000 7fd0d9400640 1 -- 172.21.15.28:0/4075170547 wait complete. 2024-03-21T08:43:18.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-21T08:43:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.438+0000 7f2822e00640 1 Processor -- start 2024-03-21T08:43:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.438+0000 7f2822e00640 1 -- start start 2024-03-21T08:43:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2822e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 0x7f281c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2822e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c0b2d20 con 0x7f281c1442b0 2024-03-21T08:43:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2822e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c0b3000 con 0x7f281c071a80 2024-03-21T08:43:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2822e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0b2ad0 con 0x7f281c144690 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f281c1442b0 0x7f281c0aebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43190/0 (socket says 172.21.15.28:43190) 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281be00640 1 -- 172.21.15.28:0/574449314 learned_addr learned my addr 172.21.15.28:0/574449314 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2813400640 1 --2- 172.21.15.28:0/574449314 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 0x7f281c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725752817 0 0) 0x7f281c0b2d20 con 0x7f281c1442b0 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800003530 con 0x7f281c1442b0 2024-03-21T08:43:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256514795 0 0) 0x7f281c0b3000 con 0x7f281c071a80 2024-03-21T08:43:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281c0b2d20 con 0x7f281c071a80 2024-03-21T08:43:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2813400640 1 -- 172.21.15.28:0/574449314 >> v1:172.21.15.151:6789/0 conn(0x7f281c071a80 legacy=0x7f281c06daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2813400640 1 -- 172.21.15.28:0/574449314 >> v1:172.21.15.28:6789/0 conn(0x7f281c1442b0 legacy=0x7f281c0aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.442+0000 7f2813400640 1 -- 172.21.15.28:0/574449314 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0b32e0 con 0x7f281c144690 2024-03-21T08:43:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2813400640 1 --2- 172.21.15.28:0/574449314 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 0x7f281c0b3dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f281c0b2c50 tx=0x7f280401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c33e10ab329135b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28040aa5f0 con 0x7f281c144690 2024-03-21T08:43:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28040af070 con 0x7f281c144690 2024-03-21T08:43:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f281b400640 1 -- 172.21.15.28:0/574449314 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28040aa990 con 0x7f281c144690 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 -- 172.21.15.28:0/574449314 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 msgr2=0x7f281c0b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 --2- 172.21.15.28:0/574449314 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 0x7f281c0b3dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f281c0b2c50 tx=0x7f280401c490 comp rx=0 tx=0).stop 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 -- 172.21.15.28:0/574449314 shutdown_connections 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 --2- 172.21.15.28:0/574449314 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c144690 0x7f281c0b3dc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 -- 172.21.15.28:0/574449314 >> 172.21.15.28:0/574449314 conn(0x7f281c0674f0 msgr2=0x7f281c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 -- 172.21.15.28:0/574449314 shutdown_connections 2024-03-21T08:43:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 -- 172.21.15.28:0/574449314 wait complete. 2024-03-21T08:43:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.446+0000 7f2822e00640 1 Processor -- start 2024-03-21T08:43:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- start start 2024-03-21T08:43:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c071a80 0x7f281c15d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f281c144690 0x7f281c13bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0722a0 con 0x7f281c1442b0 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c16bb20 con 0x7f281c144690 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c15f500 con 0x7f281c071a80 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42474/0 (socket says 172.21.15.28:42474) 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 -- 172.21.15.28:0/2279741129 learned_addr learned my addr 172.21.15.28:0/2279741129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f281be00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c071a80 0x7f281c15d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2820e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f281c144690 0x7f281c13bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 -- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c071a80 msgr2=0x7f281c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c071a80 0x7f281c15d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 -- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f281c144690 msgr2=0x7f281c13bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f281c144690 0x7f281c13bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 -- 172.21.15.28:0/2279741129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c15f7e0 con 0x7f281c1442b0 2024-03-21T08:43:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2813400640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2804000c10 tx=0x7f280401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28040be410 con 0x7f281c1442b0 2024-03-21T08:43:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28040be5b0 con 0x7f281c1442b0 2024-03-21T08:43:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28040be960 con 0x7f281c1442b0 2024-03-21T08:43:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c15fac0 con 0x7f281c1442b0 2024-03-21T08:43:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c15dbc0 con 0x7f281c1442b0 2024-03-21T08:43:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.450+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c073430 con 0x7f281c1442b0 2024-03-21T08:43:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.454+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f28040bc020 con 0x7f281c1442b0 2024-03-21T08:43:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.454+0000 7f2818c00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 0x7f27f8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.454+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2804120780 con 0x7f281c1442b0 2024-03-21T08:43:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.454+0000 7f281be00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 0x7f27f8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.454+0000 7f281be00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 0x7f27f8069fe0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f280c000f70 tx=0x7f280c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.458+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f28040160b0 con 0x7f281c1442b0 2024-03-21T08:43:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f281c04f4f0 con 0x7f281c1442b0 2024-03-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2818c00640 1 -- 172.21.15.28:0/2279741129 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v42) v1 ==== 97+0+11 (secure 0 0 0) 0x7f28040f09e0 con 0x7f281c1442b0 2024-03-21T08:43:18.628 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 msgr2=0x7f27f8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 0x7f27f8069fe0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f280c000f70 tx=0x7f280c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 msgr2=0x7f281c160290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2804000c10 tx=0x7f280401acd0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 shutdown_connections 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f281c144690 0x7f281c13bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f281c1442b0 0x7f281c160290 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f27f8067b20 0x7f27f8069fe0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 --2- 172.21.15.28:0/2279741129 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f281c071a80 0x7f281c15d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 >> 172.21.15.28:0/2279741129 conn(0x7f281c0674f0 msgr2=0x7f281c06daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 shutdown_connections 2024-03-21T08:43:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.626+0000 7f2822e00640 1 -- 172.21.15.28:0/2279741129 wait complete. 2024-03-21T08:43:18.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:18.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 Processor -- start 2024-03-21T08:43:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 -- start start 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbe2415d740 0x7fbe2415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415db20 con 0x7fbe2406e1b0 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415b700 con 0x7fbe24072920 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe29400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2406e770 con 0x7fbe2415d740 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe19e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe2406e1b0 0x7fbe24072210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43198/0 (socket says 172.21.15.28:43198) 2024-03-21T08:43:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe19e00640 1 -- 172.21.15.28:0/2804897686 learned_addr learned my addr 172.21.15.28:0/2804897686 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe22800640 1 --2- 172.21.15.28:0/2804897686 >> v2:172.21.15.151:3300/0 conn(0x7fbe2415d740 0x7fbe2415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909023539 0 0) 0x7fbe2415db20 con 0x7fbe2406e1b0 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08003530 con 0x7fbe2406e1b0 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624246257 0 0) 0x7fbe08003530 con 0x7fbe2406e1b0 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 >> v2:172.21.15.151:3300/0 conn(0x7fbe2415d740 msgr2=0x7fbe2415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 --2- 172.21.15.28:0/2804897686 >> v2:172.21.15.151:3300/0 conn(0x7fbe2415d740 0x7fbe2415b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 >> v1:172.21.15.151:6789/0 conn(0x7fbe24072920 legacy=0x7fbe2415ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2406ea50 con 0x7fbe2406e1b0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fbe0c001680 con 0x7fbe2406e1b0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.810+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 >> v1:172.21.15.28:6789/0 conn(0x7fbe2406e1b0 legacy=0x7fbe24072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe080039c0 0x7fbe08003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe080043c0 0x7fbe08004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 0x7fbe0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406e770 con 0x7fbe08005090 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0800bff0 con 0x7fbe080039c0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0800c1f0 con 0x7fbe080043c0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 0x7fbe0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe22800640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe080039c0 0x7fbe08003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe19e00640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe080043c0 0x7fbe08004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 -- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe080043c0 msgr2=0x7fbe08004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe080043c0 0x7fbe08004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 -- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe080039c0 msgr2=0x7fbe08003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe080039c0 0x7fbe08003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 -- 172.21.15.28:0/2804897686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08003530 con 0x7fbe08005090 2024-03-21T08:43:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 -- 172.21.15.28:0/2804897686 reap_dead start 2024-03-21T08:43:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe23200640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 0x7fbe0800b9d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbe24154e00 tx=0x7fbe1c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeca1fff74cfbed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c001c10 con 0x7fbe08005090 2024-03-21T08:43:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c003ba0 con 0x7fbe08005090 2024-03-21T08:43:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.814+0000 7fbe21e00640 1 -- 172.21.15.28:0/2804897686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0050d0 con 0x7fbe08005090 2024-03-21T08:43:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 msgr2=0x7fbe0800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 0x7fbe0800b9d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbe24154e00 tx=0x7fbe1c01c8c0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- 172.21.15.28:0/2804897686 shutdown_connections 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 --2- 172.21.15.28:0/2804897686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08005090 0x7fbe0800b9d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- 172.21.15.28:0/2804897686 >> 172.21.15.28:0/2804897686 conn(0x7fbe240cc230 msgr2=0x7fbe240cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- 172.21.15.28:0/2804897686 shutdown_connections 2024-03-21T08:43:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- 172.21.15.28:0/2804897686 wait complete. 2024-03-21T08:43:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 Processor -- start 2024-03-21T08:43:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- start start 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe08005090 0x7fbe24143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 0x7fbe241f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406f2f0 con 0x7fbe241527b0 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406f010 con 0x7fbe241f6ac0 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe29400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2415b1c0 con 0x7fbe08005090 2024-03-21T08:43:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe19e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe19e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42492/0 (socket says 172.21.15.28:42492) 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe19e00640 1 -- 172.21.15.28:0/864356830 learned_addr learned my addr 172.21.15.28:0/864356830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe22800640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe08005090 0x7fbe24143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.818+0000 7fbe23200640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 0x7fbe241f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 -- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe08005090 msgr2=0x7fbe24143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe08005090 0x7fbe24143d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 -- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 msgr2=0x7fbe241f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 0x7fbe241f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 -- 172.21.15.28:0/864356830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe24152fe0 con 0x7fbe241527b0 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe23200640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 0x7fbe241f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe19e00640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c012490 tx=0x7fbe0c012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c01c070 con 0x7fbe241527b0 2024-03-21T08:43:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c014770 con 0x7fbe241527b0 2024-03-21T08:43:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c014b60 con 0x7fbe241527b0 2024-03-21T08:43:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe241532c0 con 0x7fbe241527b0 2024-03-21T08:43:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe24153880 con 0x7fbe241527b0 2024-03-21T08:43:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2404f410 con 0x7fbe241527b0 2024-03-21T08:43:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbe0c014d00 con 0x7fbe241527b0 2024-03-21T08:43:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 0x7fbe0006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe0c088a30 con 0x7fbe241527b0 2024-03-21T08:43:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.822+0000 7fbe22800640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 0x7fbe0006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.826+0000 7fbe22800640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 0x7fbe0006a010 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400ac50 tx=0x7fbe1400a2a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:18.826+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbe0c058af0 con 0x7fbe241527b0 2024-03-21T08:43:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe240cd140 con 0x7fbe241527b0 2024-03-21T08:43:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe1b400640 1 -- 172.21.15.28:0/864356830 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fbe0c05cb90 con 0x7fbe241527b0 2024-03-21T08:43:19.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:19.029 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:44.201277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:43:19.029 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-03-21T08:43:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 msgr2=0x7fbe0006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 0x7fbe0006a010 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400ac50 tx=0x7fbe1400a2a0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 msgr2=0x7fbe2413d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c012490 tx=0x7fbe0c012950 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 shutdown_connections 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbe241f6ac0 0x7fbe241f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe241527b0 0x7fbe2413d2a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbe00067b50 0x7fbe0006a010 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 --2- 172.21.15.28:0/864356830 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbe08005090 0x7fbe24143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 >> 172.21.15.28:0/864356830 conn(0x7fbe240cc230 msgr2=0x7fbe24158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 shutdown_connections 2024-03-21T08:43:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.026+0000 7fbe29400640 1 -- 172.21.15.28:0/864356830 wait complete. 2024-03-21T08:43:19.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:19.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 Processor -- start 2024-03-21T08:43:19.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 -- start start 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c06c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c06e0c0 con 0x7f9a9c15dca0 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0b4140 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa1800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c06e3a0 con 0x7f9a9c0ad390 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42504/0 (socket says 172.21.15.28:42504) 2024-03-21T08:43:19.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 -- 172.21.15.28:0/2924782066 learned_addr learned my addr 172.21.15.28:0/2924782066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9aa0e00640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 -- 172.21.15.28:0/2924782066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 msgr2=0x7f9a9c06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c06c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 -- 172.21.15.28:0/2924782066 >> v1:172.21.15.28:6789/0 conn(0x7f9a9c15dca0 legacy=0x7f9a9c0b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.190+0000 7f9a93e00640 1 -- 172.21.15.28:0/2924782066 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c06e680 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009870 tx=0x7f9a8401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201e0e4129fc0608 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9a9b400640 1 -- 172.21.15.28:0/2924782066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840ab640 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9a9b400640 1 -- 172.21.15.28:0/2924782066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a840b0070 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9a9b400640 1 -- 172.21.15.28:0/2924782066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840aba00 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9aa1800640 1 -- 172.21.15.28:0/2924782066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 msgr2=0x7f9a9c0ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.194+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009870 tx=0x7f9a8401c530 comp rx=0 tx=0).stop 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- 172.21.15.28:0/2924782066 shutdown_connections 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c06c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2924782066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c0ace50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- 172.21.15.28:0/2924782066 >> 172.21.15.28:0/2924782066 conn(0x7f9a9c072080 msgr2=0x7f9a9c072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:19.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- 172.21.15.28:0/2924782066 shutdown_connections 2024-03-21T08:43:19.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- 172.21.15.28:0/2924782066 wait complete. 2024-03-21T08:43:19.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 Processor -- start 2024-03-21T08:43:19.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- start start 2024-03-21T08:43:19.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c14b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 0x7f9a9c148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c16ab60 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c06f8a0 con 0x7f9a9c15dca0 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa1800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c16ace0 con 0x7f9a9c0ad390 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42520/0 (socket says 172.21.15.28:42520) 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 -- 172.21.15.28:0/2760251712 learned_addr learned my addr 172.21.15.28:0/2760251712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a9be00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c14b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa0e00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 0x7f9a9c148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 -- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 msgr2=0x7f9a9c14b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c14b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 -- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 msgr2=0x7f9a9c148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 0x7f9a9c148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 -- 172.21.15.28:0/2760251712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c14b7a0 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9aa0e00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 0x7f9a9c148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.198+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9a8401abe0 tx=0x7f9a8401ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840be3f0 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a840be590 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c14a1c0 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840be980 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c14a780 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c075270 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9a840b43f0 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.202+0000 7f9a98c00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 0x7f9a78069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.206+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9a84120300 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.206+0000 7f9a9be00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 0x7f9a78069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.206+0000 7f9a9be00640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 0x7f9a78069fe0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c001810 tx=0x7f9a8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.210+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9a84016150 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.370+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a9c072f20 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9a98c00640 1 -- 172.21.15.28:0/2760251712 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7f9a840f03c0 con 0x7f9a9c0b2e70 2024-03-21T08:43:19.376 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:19.377 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 msgr2=0x7f9a78069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 0x7f9a78069fe0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c001810 tx=0x7f9a8c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 msgr2=0x7f9a9c149990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9a8401abe0 tx=0x7f9a8401ad70 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 shutdown_connections 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9a9c15dca0 0x7f9a9c148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0b2e70 0x7f9a9c149990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9a78067b20 0x7f9a78069fe0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 --2- 172.21.15.28:0/2760251712 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9a9c0ad390 0x7f9a9c14b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 >> 172.21.15.28:0/2760251712 conn(0x7f9a9c072080 msgr2=0x7f9a9c15cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 shutdown_connections 2024-03-21T08:43:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.374+0000 7f9aa1800640 1 -- 172.21.15.28:0/2760251712 wait complete. 2024-03-21T08:43:19.398 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-21T08:43:19.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:19.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 Processor -- start 2024-03-21T08:43:19.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 -- start start 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74a015bbe0 0x7f74a015f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f74a00cf970 0x7f74a00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a0160590 con 0x7f74a015fb50 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a015ffd0 con 0x7f74a015bbe0 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f74a4e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a01602b0 con 0x7f74a00cf970 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74a015bbe0 0x7f74a015f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74a015bbe0 0x7f74a015f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42522/0 (socket says 172.21.15.28:42522) 2024-03-21T08:43:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f7497e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f74a015fb50 0x7f74a00cedb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43214/0 (socket says 172.21.15.28:43214) 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 -- 172.21.15.28:0/335191 learned_addr learned my addr 172.21.15.28:0/335191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749e800640 1 --2- 172.21.15.28:0/335191 >> v2:172.21.15.151:3300/0 conn(0x7f74a00cf970 0x7f74a00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 -- 172.21.15.28:0/335191 >> v2:172.21.15.151:3300/0 conn(0x7f74a00cf970 msgr2=0x7f74a00cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 --2- 172.21.15.28:0/335191 >> v2:172.21.15.151:3300/0 conn(0x7f74a00cf970 0x7f74a00cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 -- 172.21.15.28:0/335191 >> v1:172.21.15.28:6789/0 conn(0x7f74a015fb50 legacy=0x7f74a00cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.538+0000 7f749de00640 1 -- 172.21.15.28:0/335191 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a00ce5b0 con 0x7f74a015bbe0 2024-03-21T08:43:19.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.542+0000 7f749de00640 1 --2- 172.21.15.28:0/335191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015bbe0 0x7f74a015f610 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f74a0072e70 tx=0x7f749001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e4718a922512ba server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749d400640 1 -- 172.21.15.28:0/335191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74900aa670 con 0x7f74a015bbe0 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749d400640 1 -- 172.21.15.28:0/335191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74900af070 con 0x7f74a015bbe0 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749d400640 1 -- 172.21.15.28:0/335191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74900aaa30 con 0x7f74a015bbe0 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- 172.21.15.28:0/335191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015bbe0 msgr2=0x7f74a015f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- 172.21.15.28:0/335191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015bbe0 0x7f74a015f610 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f74a0072e70 tx=0x7f749001c4f0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- 172.21.15.28:0/335191 shutdown_connections 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- 172.21.15.28:0/335191 >> v2:172.21.15.151:3300/0 conn(0x7f74a00cf970 0x7f74a00cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- 172.21.15.28:0/335191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015bbe0 0x7f74a015f610 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- 172.21.15.28:0/335191 >> 172.21.15.28:0/335191 conn(0x7f74a0099a30 msgr2=0x7f74a0099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- 172.21.15.28:0/335191 shutdown_connections 2024-03-21T08:43:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- 172.21.15.28:0/335191 wait complete. 2024-03-21T08:43:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 Processor -- start 2024-03-21T08:43:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- start start 2024-03-21T08:43:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 0x7f74a0155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74a015bbe0 0x7f74a00c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a01689b0 con 0x7f74a015fb50 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00ce060 con 0x7f74a00cf970 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f74a4e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0168b30 con 0x7f74a015bbe0 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42538/0 (socket says 172.21.15.28:42538) 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.546+0000 7f749e800640 1 -- 172.21.15.28:0/3371377414 learned_addr learned my addr 172.21.15.28:0/3371377414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749de00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 0x7f74a0155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7497e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74a015bbe0 0x7f74a00c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 -- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74a015bbe0 msgr2=0x7f74a00c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74a015bbe0 0x7f74a00c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 -- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 msgr2=0x7f74a0155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 0x7f74a0155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 -- 172.21.15.28:0/3371377414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0155c70 con 0x7f74a015fb50 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749de00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 0x7f74a0155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f749e800640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7498009d30 tx=0x7f749800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498018020 con 0x7f74a015fb50 2024-03-21T08:43:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498014480 con 0x7f74a015fb50 2024-03-21T08:43:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a00b5380 con 0x7f74a015fb50 2024-03-21T08:43:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498014870 con 0x7f74a015fb50 2024-03-21T08:43:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a00b5940 con 0x7f74a015fb50 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a004f410 con 0x7f74a015fb50 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f749801b020 con 0x7f74a015fb50 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.550+0000 7f7496000640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 0x7f747c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.554+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7498086c20 con 0x7f74a015fb50 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.554+0000 7f749de00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 0x7f747c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:19.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.554+0000 7f749de00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 0x7f747c069fe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f74900ae000 tx=0x7f74900a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.554+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7498022070 con 0x7f74a015fb50 2024-03-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.894+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74a00cd420 con 0x7f74a015fb50 2024-03-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.894+0000 7f7496000640 1 -- 172.21.15.28:0/3371377414 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f74a0155c70 con 0x7f74a015fb50 2024-03-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:44.201277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-03-21T08:43:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 msgr2=0x7f747c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 0x7f747c069fe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f74900ae000 tx=0x7f74900a7000 comp rx=0 tx=0).stop 2024-03-21T08:43:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 msgr2=0x7f74a00b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7498009d30 tx=0x7f749800c490 comp rx=0 tx=0).stop 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 shutdown_connections 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74a015fb50 0x7f74a00b3470 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f747c067b20 0x7f747c069fe0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f74a015bbe0 0x7f74a00c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 --2- 172.21.15.28:0/3371377414 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f74a00cf970 0x7f74a0155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 >> 172.21.15.28:0/3371377414 conn(0x7f74a0099a30 msgr2=0x7f74a006f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 shutdown_connections 2024-03-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:19.898+0000 7f74a4e00640 1 -- 172.21.15.28:0/3371377414 wait complete. 2024-03-21T08:43:19.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.058+0000 7fe145000640 1 Processor -- start 2024-03-21T08:43:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.058+0000 7fe145000640 1 -- start start 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe145000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 0x7fe1400cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe145000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 0x7fe14006b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe145000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14006bad0 con 0x7fe14015c680 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe145000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400cffe0 con 0x7fe14006b170 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe145000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400ceaa0 con 0x7fe1400ce4c0 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe14015c680 0x7fe14015ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43222/0 (socket says 172.21.15.28:43222) 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13de00640 1 -- 172.21.15.28:0/2560100018 learned_addr learned my addr 172.21.15.28:0/2560100018 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13e800640 1 --2- 172.21.15.28:0/2560100018 >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 0x7fe14006b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe137e00640 1 --2- 172.21.15.28:0/2560100018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 0x7fe1400cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13d400640 1 -- 172.21.15.28:0/2560100018 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274589155 0 0) 0x7fe14006bad0 con 0x7fe14015c680 2024-03-21T08:43:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13d400640 1 -- 172.21.15.28:0/2560100018 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe124003530 con 0x7fe14015c680 2024-03-21T08:43:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe137e00640 1 -- 172.21.15.28:0/2560100018 >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 msgr2=0x7fe14006b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe137e00640 1 --2- 172.21.15.28:0/2560100018 >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 0x7fe14006b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe137e00640 1 -- 172.21.15.28:0/2560100018 >> v1:172.21.15.28:6789/0 conn(0x7fe14015c680 legacy=0x7fe14015ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe137e00640 1 -- 172.21.15.28:0/2560100018 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1400ced80 con 0x7fe1400ce4c0 2024-03-21T08:43:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.062+0000 7fe13e800640 1 --2- 172.21.15.28:0/2560100018 >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 0x7fe14006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe137e00640 1 --2- 172.21.15.28:0/2560100018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 0x7fe1400cfaa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1280097c0 tx=0x7fe12801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46408f635832eec server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe13d400640 1 -- 172.21.15.28:0/2560100018 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280b0070 con 0x7fe1400ce4c0 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe13d400640 1 -- 172.21.15.28:0/2560100018 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1280ab600 con 0x7fe1400ce4c0 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe13d400640 1 -- 172.21.15.28:0/2560100018 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1280ab9b0 con 0x7fe1400ce4c0 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- 172.21.15.28:0/2560100018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 msgr2=0x7fe1400cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 --2- 172.21.15.28:0/2560100018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 0x7fe1400cfaa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe1280097c0 tx=0x7fe12801c410 comp rx=0 tx=0).stop 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- 172.21.15.28:0/2560100018 shutdown_connections 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 --2- 172.21.15.28:0/2560100018 >> v2:172.21.15.151:3300/0 conn(0x7fe14006b170 0x7fe14006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 --2- 172.21.15.28:0/2560100018 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe1400ce4c0 0x7fe1400cfaa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- 172.21.15.28:0/2560100018 >> 172.21.15.28:0/2560100018 conn(0x7fe140099a30 msgr2=0x7fe140099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- 172.21.15.28:0/2560100018 shutdown_connections 2024-03-21T08:43:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- 172.21.15.28:0/2560100018 wait complete. 2024-03-21T08:43:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 Processor -- start 2024-03-21T08:43:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.066+0000 7fe145000640 1 -- start start 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1400ce4c0 0x7fe140153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe14015c680 0x7fe14015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400cd500 con 0x7fe14006b170 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400cd380 con 0x7fe1400ce4c0 2024-03-21T08:43:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe140168aa0 con 0x7fe14015c680 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42546/0 (socket says 172.21.15.28:42546) 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 -- 172.21.15.28:0/1948965517 learned_addr learned my addr 172.21.15.28:0/1948965517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe137e00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1400ce4c0 0x7fe140153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13e800640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe14015c680 0x7fe14015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 -- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe14015c680 msgr2=0x7fe14015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe14015c680 0x7fe14015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 -- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1400ce4c0 msgr2=0x7fe140153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1400ce4c0 0x7fe140153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 -- 172.21.15.28:0/1948965517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14006eea0 con 0x7fe14006b170 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe13de00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c450 tx=0x7fe13000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe130017070 con 0x7fe14006b170 2024-03-21T08:43:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe130005cf0 con 0x7fe14006b170 2024-03-21T08:43:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14006f180 con 0x7fe14006b170 2024-03-21T08:43:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1300072f0 con 0x7fe14006b170 2024-03-21T08:43:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14006f740 con 0x7fe14006b170 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14004f410 con 0x7fe14006b170 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.070+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fe13001a070 con 0x7fe14006b170 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.074+0000 7fe136000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 0x7fe11c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.074+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe130085600 con 0x7fe14006b170 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.074+0000 7fe137e00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 0x7fe11c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.074+0000 7fe137e00640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 0x7fe11c06a010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1280ad000 tx=0x7fe128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.078+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe130055630 con 0x7fe14006b170 2024-03-21T08:43:20.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.274+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1400b6870 con 0x7fe14006b170 2024-03-21T08:43:20.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.274+0000 7fe136000640 1 -- 172.21.15.28:0/1948965517 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fe1300596d0 con 0x7fe14006b170 2024-03-21T08:43:20.279 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:20.279 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:42:30.068610+0000","modified":"2024-03-21T08:42:44.201277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/3938951152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3938951152},{"type":"v1","addr":"172.21.15.28:6834","nonce":3938951152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5517,"qdb_cluster":[5517]},"id":2}]} 2024-03-21T08:43:20.279 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 19 2024-03-21T08:43:20.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 msgr2=0x7fe11c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 0x7fe11c06a010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1280ad000 tx=0x7fe128008040 comp rx=0 tx=0).stop 2024-03-21T08:43:20.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 msgr2=0x7fe1401f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c450 tx=0x7fe13000c850 comp rx=0 tx=0).stop 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 shutdown_connections 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe11c067b50 0x7fe11c06a010 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fe14015c680 0x7fe14015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe1400ce4c0 0x7fe140153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 --2- 172.21.15.28:0/1948965517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe14006b170 0x7fe1401f5e70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 >> 172.21.15.28:0/1948965517 conn(0x7fe140099a30 msgr2=0x7fe14015d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 shutdown_connections 2024-03-21T08:43:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.278+0000 7fe145000640 1 -- 172.21.15.28:0/1948965517 wait complete. 2024-03-21T08:43:20.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 Processor -- start 2024-03-21T08:43:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 -- start start 2024-03-21T08:43:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fc59c15f260 0x7fc59c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc59c0cd5d0 0x7fc59c099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c0cdb50 con 0x7fc59c09a0c0 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59c0c5530 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a4800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59c0cde30 con 0x7fc59c15f260 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc59c0cd5d0 0x7fc59c099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc59c0cd5d0 0x7fc59c099b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50940/0 (socket says 172.21.15.28:50940) 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a1400640 1 -- 172.21.15.28:0/723864703 learned_addr learned my addr 172.21.15.28:0/723864703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.438+0000 7fc5a1e00640 1 --2- 172.21.15.28:0/723864703 >> v2:172.21.15.151:3300/0 conn(0x7fc59c15f260 0x7fc59c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a1400640 1 -- 172.21.15.28:0/723864703 >> v2:172.21.15.151:3300/0 conn(0x7fc59c15f260 msgr2=0x7fc59c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a1400640 1 --2- 172.21.15.28:0/723864703 >> v2:172.21.15.151:3300/0 conn(0x7fc59c15f260 0x7fc59c0cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a1400640 1 -- 172.21.15.28:0/723864703 >> v1:172.21.15.151:6789/0 conn(0x7fc59c09a0c0 legacy=0x7fc59c0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a1400640 1 -- 172.21.15.28:0/723864703 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c0c5780 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a1400640 1 --2- 172.21.15.28:0/723864703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c0cd5d0 0x7fc59c099b80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc59c0b4110 tx=0x7fc58c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbefa9249713b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a0a00640 1 -- 172.21.15.28:0/723864703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0b1070 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a0a00640 1 -- 172.21.15.28:0/723864703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c0ac600 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.442+0000 7fc5a0a00640 1 -- 172.21.15.28:0/723864703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0ac9d0 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- 172.21.15.28:0/723864703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c0cd5d0 msgr2=0x7fc59c099b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- 172.21.15.28:0/723864703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c0cd5d0 0x7fc59c099b80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc59c0b4110 tx=0x7fc58c01c490 comp rx=0 tx=0).stop 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- 172.21.15.28:0/723864703 shutdown_connections 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- 172.21.15.28:0/723864703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c0cd5d0 0x7fc59c099b80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- 172.21.15.28:0/723864703 >> v2:172.21.15.151:3300/0 conn(0x7fc59c15f260 0x7fc59c0cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- 172.21.15.28:0/723864703 >> 172.21.15.28:0/723864703 conn(0x7fc59c0b34d0 msgr2=0x7fc59c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- 172.21.15.28:0/723864703 shutdown_connections 2024-03-21T08:43:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- 172.21.15.28:0/723864703 wait complete. 2024-03-21T08:43:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 Processor -- start 2024-03-21T08:43:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- start start 2024-03-21T08:43:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc59c09a0c0 0x7fc59c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc59c0cd5d0 0x7fc59c0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c168bc0 con 0x7fc59c15f260 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c168a40 con 0x7fc59c09a0c0 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a4800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c168d40 con 0x7fc59c0cd5d0 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50950/0 (socket says 172.21.15.28:50950) 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 -- 172.21.15.28:0/2819770261 learned_addr learned my addr 172.21.15.28:0/2819770261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a1e00640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc59c09a0c0 0x7fc59c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a1400640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc59c0cd5d0 0x7fc59c0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 -- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc59c0cd5d0 msgr2=0x7fc59c0c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc59c0cd5d0 0x7fc59c0c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 -- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc59c09a0c0 msgr2=0x7fc59c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc59c09a0c0 0x7fc59c0cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.446+0000 7fc5a2800640 1 -- 172.21.15.28:0/2819770261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c1fac50 con 0x7fc59c15f260 2024-03-21T08:43:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc5a2800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c760 tx=0x7fc59800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598003b80 con 0x7fc59c15f260 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc598003d20 con 0x7fc59c15f260 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59800aa30 con 0x7fc59c15f260 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c166720 con 0x7fc59c15f260 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c0ca950 con 0x7fc59c15f260 2024-03-21T08:43:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c0c8ff0 con 0x7fc59c15f260 2024-03-21T08:43:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fc59800abd0 con 0x7fc59c15f260 2024-03-21T08:43:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 0x7fc580069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc598085ff0 con 0x7fc59c15f260 2024-03-21T08:43:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.450+0000 7fc5a1e00640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 0x7fc580069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.454+0000 7fc5a1e00640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 0x7fc580069fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc590006aa0 tx=0x7fc590008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.454+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc59c166720 con 0x7fc59c15f260 2024-03-21T08:43:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.622+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc59c04f410 con 0x7fc59c15f260 2024-03-21T08:43:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.622+0000 7fc596000640 1 -- 172.21.15.28:0/2819770261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18766 (secure 0 0 0) 0x7fc598056040 con 0x7fc59c15f260 2024-03-21T08:43:20.625 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:20.625 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:42:45.819075+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:42:27.937645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-21T08:42:29.064158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-21T08:42:43.385115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 msgr2=0x7fc580069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 0x7fc580069fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc590006aa0 tx=0x7fc590008040 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 msgr2=0x7fc59c158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c760 tx=0x7fc59800cc20 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 shutdown_connections 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc59c15f260 0x7fc59c158e40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fc580067b20 0x7fc580069fe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fc59c0cd5d0 0x7fc59c0c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 --2- 172.21.15.28:0/2819770261 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc59c09a0c0 0x7fc59c0cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 >> 172.21.15.28:0/2819770261 conn(0x7fc59c0b34d0 msgr2=0x7fc59c09a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 shutdown_connections 2024-03-21T08:43:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.626+0000 7fc5a4800640 1 -- 172.21.15.28:0/2819770261 wait complete. 2024-03-21T08:43:20.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-21T08:43:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 Processor -- start 2024-03-21T08:43:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 -- start start 2024-03-21T08:43:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f70980b0e80 0x7f70980ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f70980cea00 0x7f709808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70980cede0 con 0x7f709815bd20 2024-03-21T08:43:20.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709815fba0 con 0x7f70980b0e80 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709fe00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709815fe80 con 0x7f70980cea00 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f70980b0e80 0x7f70980ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f70980b0e80 0x7f70980ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50960/0 (socket says 172.21.15.28:50960) 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709ca00640 1 -- 172.21.15.28:0/3377263428 learned_addr learned my addr 172.21.15.28:0/3377263428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.786+0000 7f709de00640 1 --2- 172.21.15.28:0/3377263428 >> v2:172.21.15.151:3300/0 conn(0x7f70980cea00 0x7f709808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709ca00640 1 -- 172.21.15.28:0/3377263428 >> v2:172.21.15.151:3300/0 conn(0x7f70980cea00 msgr2=0x7f709808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709ca00640 1 --2- 172.21.15.28:0/3377263428 >> v2:172.21.15.151:3300/0 conn(0x7f70980cea00 0x7f709808afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709ca00640 1 -- 172.21.15.28:0/3377263428 >> v1:172.21.15.151:6789/0 conn(0x7f709815bd20 legacy=0x7f70980b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709ca00640 1 -- 172.21.15.28:0/3377263428 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098160160 con 0x7f70980b0e80 2024-03-21T08:43:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709ca00640 1 --2- 172.21.15.28:0/3377263428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70980b0e80 0x7f70980ce4c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f70980b1960 tx=0x7f708401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00fe4cd86b3a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f7097e00640 1 -- 172.21.15.28:0/3377263428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840b0070 con 0x7f70980b0e80 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f7097e00640 1 -- 172.21.15.28:0/3377263428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70840ab600 con 0x7f70980b0e80 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f7097e00640 1 -- 172.21.15.28:0/3377263428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840ab9d0 con 0x7f70980b0e80 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709fe00640 1 -- 172.21.15.28:0/3377263428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70980b0e80 msgr2=0x7f70980ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.790+0000 7f709fe00640 1 --2- 172.21.15.28:0/3377263428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70980b0e80 0x7f70980ce4c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f70980b1960 tx=0x7f708401c520 comp rx=0 tx=0).stop 2024-03-21T08:43:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 -- 172.21.15.28:0/3377263428 shutdown_connections 2024-03-21T08:43:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 --2- 172.21.15.28:0/3377263428 >> v2:172.21.15.151:3300/0 conn(0x7f70980cea00 0x7f709808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 --2- 172.21.15.28:0/3377263428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70980b0e80 0x7f70980ce4c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 -- 172.21.15.28:0/3377263428 >> 172.21.15.28:0/3377263428 conn(0x7f70980d1ab0 msgr2=0x7f70980d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 -- 172.21.15.28:0/3377263428 shutdown_connections 2024-03-21T08:43:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 -- 172.21.15.28:0/3377263428 wait complete. 2024-03-21T08:43:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 Processor -- start 2024-03-21T08:43:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.794+0000 7f709fe00640 1 -- start start 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70980b0e80 0x7f709806cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 0x7f7098164e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709808be90 con 0x7f709815bd20 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709808bd10 con 0x7f70980cea00 2024-03-21T08:43:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7098168d10 con 0x7f70980b0e80 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50964/0 (socket says 172.21.15.28:50964) 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 -- 172.21.15.28:0/1254513857 learned_addr learned my addr 172.21.15.28:0/1254513857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709ca00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 0x7f7098164e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709d400640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70980b0e80 0x7f709806cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 -- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70980b0e80 msgr2=0x7f709806cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70980b0e80 0x7f709806cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 -- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 msgr2=0x7f7098164e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 0x7f7098164e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 -- 172.21.15.28:0/1254513857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098165de0 con 0x7f709815bd20 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709ca00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 0x7f7098164e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709de00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7090007e70 tx=0x7f709000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090014620 con 0x7f709815bd20 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090019070 con 0x7f709815bd20 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090014a20 con 0x7f709815bd20 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7098165540 con 0x7f709815bd20 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7098165b00 con 0x7f709815bd20 2024-03-21T08:43:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.798+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709804f4f0 con 0x7f709815bd20 2024-03-21T08:43:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.802+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7090014bc0 con 0x7f709815bd20 2024-03-21T08:43:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.802+0000 7f7095600640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 0x7f7078069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.802+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7090086a40 con 0x7f709815bd20 2024-03-21T08:43:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.802+0000 7f709d400640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 0x7f7078069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.802+0000 7f709d400640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 0x7f7078069fe0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f708c0015b0 tx=0x7f708c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:20.806+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f70900571d0 con 0x7f709815bd20 2024-03-21T08:43:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.002+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f70980d2830 con 0x7f709815bd20 2024-03-21T08:43:21.004 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-03-21T08:43:21.002+0000 7f91dfc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T08:43:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.002+0000 7f7095600640 1 -- 172.21.15.28:0/1254513857 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7090056ad0 con 0x7f709815bd20 2024-03-21T08:43:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 msgr2=0x7f7078069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 0x7f7078069fe0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f708c0015b0 tx=0x7f708c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 msgr2=0x7f7098166680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7090007e70 tx=0x7f709000e4b0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 shutdown_connections 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f709815bd20 0x7f7098166680 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f70980cea00 0x7f7098164e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7078067b20 0x7f7078069fe0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 --2- 172.21.15.28:0/1254513857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f70980b0e80 0x7f709806cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 >> 172.21.15.28:0/1254513857 conn(0x7f70980d1ab0 msgr2=0x7f7098157e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 shutdown_connections 2024-03-21T08:43:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.006+0000 7f709fe00640 1 -- 172.21.15.28:0/1254513857 wait complete. 2024-03-21T08:43:21.025 INFO:tasks.ceph.mds.a.smithi028.stderr: -4013> 2024-03-21T08:42:34.651+0000 7fdca0e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:43:21.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-21T08:43:21.082 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-03-21T08:43:21.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.170+0000 7fdfca000640 1 Processor -- start 2024-03-21T08:43:21.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.170+0000 7fdfca000640 1 -- start start 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfca000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdfc40d0ee0 0x7fdfc4168ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfca000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc406bd40 con 0x7fdfc415e190 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfca000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc406c020 con 0x7fdfc40d0320 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfca000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc406c300 con 0x7fdfc40d0ee0 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfbbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdfc415e190 0x7fdfc40b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59560/0 (socket says 172.21.15.28:59560) 2024-03-21T08:43:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfbbe00640 1 -- 172.21.15.28:0/3907989561 learned_addr learned my addr 172.21.15.28:0/3907989561 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3e00640 1 --2- 172.21.15.28:0/3907989561 >> v2:172.21.15.151:3300/0 conn(0x7fdfc40d0ee0 0x7fdfc4168ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278577993 0 0) 0x7fdfc406bd40 con 0x7fdfc415e190 2024-03-21T08:43:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa8003530 con 0x7fdfc415e190 2024-03-21T08:43:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005863900 0 0) 0x7fdfa8003530 con 0x7fdfc415e190 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 >> v2:172.21.15.151:3300/0 conn(0x7fdfc40d0ee0 msgr2=0x7fdfc4168ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 --2- 172.21.15.28:0/3907989561 >> v2:172.21.15.151:3300/0 conn(0x7fdfc40d0ee0 0x7fdfc4168ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 >> v1:172.21.15.151:6789/0 conn(0x7fdfc40d0320 legacy=0x7fdfc40d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc406c5e0 con 0x7fdfc415e190 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fdfac001680 con 0x7fdfc415e190 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 >> v1:172.21.15.28:6789/0 conn(0x7fdfc415e190 legacy=0x7fdfc40b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 0x7fdfa8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa8004810 0x7fdfa8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfa80051d0 0x7fdfa800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc406c300 con 0x7fdfa80039c0 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa800c170 con 0x7fdfa80051d0 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa800c370 con 0x7fdfa8004810 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 0x7fdfa8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfbbe00640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa8004810 0x7fdfa8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3e00640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfa80051d0 0x7fdfa800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 -- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa8004810 msgr2=0x7fdfa8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa8004810 0x7fdfa8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 -- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfa80051d0 msgr2=0x7fdfa800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfa80051d0 0x7fdfa800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3400640 1 -- 172.21.15.28:0/3907989561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa8003530 con 0x7fdfa80039c0 2024-03-21T08:43:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.174+0000 7fdfc3e00640 1 -- 172.21.15.28:0/3907989561 reap_dead start 2024-03-21T08:43:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfc3400640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 0x7fdfa8003de0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4001780 tx=0x7fdfb401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a273408def5bc539 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40013a0 con 0x7fdfa80039c0 2024-03-21T08:43:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb40052f0 con 0x7fdfa80039c0 2024-03-21T08:43:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfc2a00640 1 -- 172.21.15.28:0/3907989561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4003840 con 0x7fdfa80039c0 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 -- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 msgr2=0x7fdfa8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 0x7fdfa8003de0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4001780 tx=0x7fdfb401c400 comp rx=0 tx=0).stop 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 -- 172.21.15.28:0/3907989561 shutdown_connections 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 --2- 172.21.15.28:0/3907989561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa80039c0 0x7fdfa8003de0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 -- 172.21.15.28:0/3907989561 >> 172.21.15.28:0/3907989561 conn(0x7fdfc4099a30 msgr2=0x7fdfc40b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 -- 172.21.15.28:0/3907989561 shutdown_connections 2024-03-21T08:43:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.178+0000 7fdfca000640 1 -- 172.21.15.28:0/3907989561 wait complete. 2024-03-21T08:43:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 Processor -- start 2024-03-21T08:43:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- start start 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa80039c0 0x7fdfc4157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc41f83c0 0x7fdfc4160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40b0230 con 0x7fdfc4160a90 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40b00b0 con 0x7fdfc41f83c0 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40b03b0 con 0x7fdfa80039c0 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51004/0 (socket says 172.21.15.28:51004) 2024-03-21T08:43:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 -- 172.21.15.28:0/1805173740 learned_addr learned my addr 172.21.15.28:0/1805173740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3400640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa80039c0 0x7fdfc4157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfbbe00640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc41f83c0 0x7fdfc4160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 -- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa80039c0 msgr2=0x7fdfc4157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa80039c0 0x7fdfc4157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 -- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc41f83c0 msgr2=0x7fdfc4160550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc41f83c0 0x7fdfc4160550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 -- 172.21.15.28:0/1805173740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc415e910 con 0x7fdfc4160a90 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfc3e00640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4072e20 tx=0x7fdfbc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00fdb0 con 0x7fdfc4160a90 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc01d070 con 0x7fdfc4160a90 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc015670 con 0x7fdfc4160a90 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc415ebf0 con 0x7fdfc4160a90 2024-03-21T08:43:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc415f1b0 con 0x7fdfc4160a90 2024-03-21T08:43:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.182+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc404f4f0 con 0x7fdfc4160a90 2024-03-21T08:43:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.186+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fdfbc015810 con 0x7fdfc4160a90 2024-03-21T08:43:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.186+0000 7fdfbb400640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 0x7fdfa0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.186+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6582+0+0 (secure 0 0 0) 0x7fdfbc086a60 con 0x7fdfc4160a90 2024-03-21T08:43:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.186+0000 7fdfc3400640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 0x7fdfa0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.186+0000 7fdfc3400640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 0x7fdfa0069fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4001750 tx=0x7fdfb40011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.190+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdfbc010040 con 0x7fdfc4160a90 2024-03-21T08:43:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:21.390+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdfc40d0100 con 0x7fdfc4160a90 2024-03-21T08:43:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.006+0000 7fdfbb400640 1 -- 172.21.15.28:0/1805173740 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdfbc056a10 con 0x7fdfc4160a90 2024-03-21T08:43:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 msgr2=0x7fdfa0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 0x7fdfa0069fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4001750 tx=0x7fdfb40011a0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 msgr2=0x7fdfc41faf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4072e20 tx=0x7fdfbc00f580 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 shutdown_connections 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfc4160a90 0x7fdfc41faf70 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdfc41f83c0 0x7fdfc4160550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdfa0067b20 0x7fdfa0069fe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 --2- 172.21.15.28:0/1805173740 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdfa80039c0 0x7fdfc4157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 >> 172.21.15.28:0/1805173740 conn(0x7fdfc4099a30 msgr2=0x7fdfc40b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 shutdown_connections 2024-03-21T08:43:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.010+0000 7fdfca000640 1 -- 172.21.15.28:0/1805173740 wait complete. 2024-03-21T08:43:22.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-21T08:43:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 Processor -- start 2024-03-21T08:43:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 -- start start 2024-03-21T08:43:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f80780ce6d0 0x7f807806e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 0x7f807815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807806da60 con 0x7f807815c150 2024-03-21T08:43:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807815cc80 con 0x7f80780ce6d0 2024-03-21T08:43:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f8080800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807815cf60 con 0x7f8078072130 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807e800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 0x7f807815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807d400640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f807815c150 0x7f8078071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:36832/0 (socket says 172.21.15.28:36832) 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807d400640 1 -- 172.21.15.28:0/814132944 learned_addr learned my addr 172.21.15.28:0/814132944 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807de00640 1 --2- 172.21.15.28:0/814132944 >> v2:172.21.15.151:3300/0 conn(0x7f80780ce6d0 0x7f807806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807ca00640 1 -- 172.21.15.28:0/814132944 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663992 0 0) 0x7f807806da60 con 0x7f807815c150 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807ca00640 1 -- 172.21.15.28:0/814132944 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8060003530 con 0x7f807815c150 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807de00640 1 -- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 msgr2=0x7f807815c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807de00640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 0x7f807815c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807de00640 1 -- 172.21.15.28:0/814132944 >> v1:172.21.15.151:6789/0 conn(0x7f807815c150 legacy=0x7f8078071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807de00640 1 -- 172.21.15.28:0/814132944 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807806dd20 con 0x7f80780ce6d0 2024-03-21T08:43:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.182+0000 7f807e800640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 0x7f807815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f807de00640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f807806e520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f806c0097c0 tx=0x7f806c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a1aa210eca35168 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f807ca00640 1 -- 172.21.15.28:0/814132944 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0b0070 con 0x7f80780ce6d0 2024-03-21T08:43:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f807ca00640 1 -- 172.21.15.28:0/814132944 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c0ab600 con 0x7f80780ce6d0 2024-03-21T08:43:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f807ca00640 1 -- 172.21.15.28:0/814132944 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0ab9d0 con 0x7f80780ce6d0 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 -- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 msgr2=0x7f807806e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f807806e520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f806c0097c0 tx=0x7f806c01c410 comp rx=0 tx=0).stop 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 -- 172.21.15.28:0/814132944 shutdown_connections 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8078072130 0x7f807815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 --2- 172.21.15.28:0/814132944 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f807806e520 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 -- 172.21.15.28:0/814132944 >> 172.21.15.28:0/814132944 conn(0x7f80780674f0 msgr2=0x7f807806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 -- 172.21.15.28:0/814132944 shutdown_connections 2024-03-21T08:43:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.186+0000 7f8080800640 1 -- 172.21.15.28:0/814132944 wait complete. 2024-03-21T08:43:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 Processor -- start 2024-03-21T08:43:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- start start 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f8078150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f807815c150 0x7f807815e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80780ccd90 con 0x7f8078072130 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80780ccc10 con 0x7f80780ce6d0 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80780ccf10 con 0x7f807815c150 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51016/0 (socket says 172.21.15.28:51016) 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 -- 172.21.15.28:0/3661686450 learned_addr learned my addr 172.21.15.28:0/3661686450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807d400640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f8078150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807e800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f807815c150 0x7f807815e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 -- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f807815c150 msgr2=0x7f807815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f807815c150 0x7f807815e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 -- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 msgr2=0x7f8078150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f8078150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 -- 172.21.15.28:0/3661686450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807815d4d0 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f807de00640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f806c01a040 tx=0x7f806c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0b8020 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c0b4410 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807815d7b0 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0b0040 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.190+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807815dd70 con 0x7f8078072130 2024-03-21T08:43:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.194+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8078068110 con 0x7f8078072130 2024-03-21T08:43:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.194+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f806c0bb020 con 0x7f8078072130 2024-03-21T08:43:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.194+0000 7f8072000640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 0x7f805c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.194+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6582+0+0 (secure 0 0 0) 0x7f806c11fcb0 con 0x7f8078072130 2024-03-21T08:43:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.194+0000 7f807d400640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 0x7f805c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.198+0000 7f807d400640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 0x7f805c069fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8068007fd0 tx=0x7f8068009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.198+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f806c0c2030 con 0x7f8078072130 2024-03-21T08:43:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:22.366+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8078072b80 con 0x7f8078072130 2024-03-21T08:43:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.006+0000 7f8072000640 1 -- 172.21.15.28:0/3661686450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 181+0+0 (secure 0 0 0) 0x7f806c016090 con 0x7f8078072130 2024-03-21T08:43:23.011 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-03-21T08:43:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 msgr2=0x7f805c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 0x7f805c069fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8068007fd0 tx=0x7f8068009040 comp rx=0 tx=0).stop 2024-03-21T08:43:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 msgr2=0x7f8078146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f806c01a040 tx=0x7f806c0191b0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 shutdown_connections 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f805c067b20 0x7f805c069fe0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f807815c150 0x7f807815e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80780ce6d0 0x7f8078150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 --2- 172.21.15.28:0/3661686450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8078072130 0x7f8078146e80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 >> 172.21.15.28:0/3661686450 conn(0x7f80780674f0 msgr2=0x7f80780b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 shutdown_connections 2024-03-21T08:43:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.010+0000 7f8080800640 1 -- 172.21.15.28:0/3661686450 wait complete. 2024-03-21T08:43:23.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-21T08:43:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 Processor -- start 2024-03-21T08:43:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 -- start start 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f8f7415e050 0x7f8f7406d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8f74099c20 0x7f8f7409a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f740d1600 con 0x7f8f740ccf40 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7409a5c0 con 0x7f8f74099c20 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f79e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f740ccab0 con 0x7f8f7415e050 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8f74099c20 0x7f8f7409a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8f74099c20 0x7f8f7409a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51020/0 (socket says 172.21.15.28:51020) 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 -- 172.21.15.28:0/250843699 learned_addr learned my addr 172.21.15.28:0/250843699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f72a00640 1 --2- 172.21.15.28:0/250843699 >> v2:172.21.15.151:3300/0 conn(0x7f8f7415e050 0x7f8f7406d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 -- 172.21.15.28:0/250843699 >> v2:172.21.15.151:3300/0 conn(0x7f8f7415e050 msgr2=0x7f8f7406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 --2- 172.21.15.28:0/250843699 >> v2:172.21.15.151:3300/0 conn(0x7f8f7415e050 0x7f8f7406d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 -- 172.21.15.28:0/250843699 >> v1:172.21.15.151:6789/0 conn(0x7f8f740ccf40 legacy=0x7f8f740cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.186+0000 7f8f6a000640 1 -- 172.21.15.28:0/250843699 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740cc180 con 0x7f8f74099c20 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f6a000640 1 --2- 172.21.15.28:0/250843699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74099c20 0x7f8f7409a080 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8f7406f950 tx=0x7f8f5c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d19739f8fa167a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f72000640 1 -- 172.21.15.28:0/250843699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a5730 con 0x7f8f74099c20 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f72000640 1 -- 172.21.15.28:0/250843699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0a58d0 con 0x7f8f74099c20 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f72000640 1 -- 172.21.15.28:0/250843699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a5ca0 con 0x7f8f74099c20 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 -- 172.21.15.28:0/250843699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74099c20 msgr2=0x7f8f7409a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 --2- 172.21.15.28:0/250843699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74099c20 0x7f8f7409a080 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8f7406f950 tx=0x7f8f5c01c960 comp rx=0 tx=0).stop 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 -- 172.21.15.28:0/250843699 shutdown_connections 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 --2- 172.21.15.28:0/250843699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74099c20 0x7f8f7409a080 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 --2- 172.21.15.28:0/250843699 >> v2:172.21.15.151:3300/0 conn(0x7f8f7415e050 0x7f8f7406d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 -- 172.21.15.28:0/250843699 >> 172.21.15.28:0/250843699 conn(0x7f8f740b34f0 msgr2=0x7f8f740b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 -- 172.21.15.28:0/250843699 shutdown_connections 2024-03-21T08:43:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 -- 172.21.15.28:0/250843699 wait complete. 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.190+0000 7f8f79e00640 1 Processor -- start 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- start start 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f74099c20 0x7f8f741531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f740ccf40 0x7f8f740cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740d1170 con 0x7f8f7415e050 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740d12f0 con 0x7f8f74099c20 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7416b2b0 con 0x7f8f740ccf40 2024-03-21T08:43:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51022/0 (socket says 172.21.15.28:51022) 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 -- 172.21.15.28:0/2316771750 learned_addr learned my addr 172.21.15.28:0/2316771750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f72a00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f74099c20 0x7f8f741531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f6a000640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f740ccf40 0x7f8f740cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 -- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f740ccf40 msgr2=0x7f8f740cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f740ccf40 0x7f8f740cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 -- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f74099c20 msgr2=0x7f8f741531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f74099c20 0x7f8f741531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 -- 172.21.15.28:0/2316771750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74153700 con 0x7f8f7415e050 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f73400640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00c430 tx=0x7f8f6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c005220 con 0x7f8f7415e050 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c0053c0 con 0x7f8f7415e050 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c007270 con 0x7f8f7415e050 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f740ca2e0 con 0x7f8f7415e050 2024-03-21T08:43:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f740ca8a0 con 0x7f8f7415e050 2024-03-21T08:43:23.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.194+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f74068110 con 0x7f8f7415e050 2024-03-21T08:43:23.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.198+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8f6c019020 con 0x7f8f7415e050 2024-03-21T08:43:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.198+0000 7f8f6be00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 0x7f8f50069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.198+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f8f6c085330 con 0x7f8f7415e050 2024-03-21T08:43:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.198+0000 7f8f72a00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 0x7f8f50069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.198+0000 7f8f72a00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 0x7f8f50069fe0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8f64006aa0 tx=0x7f8f64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.202+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8f6c055540 con 0x7f8f7415e050 2024-03-21T08:43:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:23.366+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8f740ced90 con 0x7f8f7415e050 2024-03-21T08:43:24.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.010+0000 7f8f6be00640 1 -- 172.21.15.28:0/2316771750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8f6c0595e0 con 0x7f8f7415e050 2024-03-21T08:43:24.013 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-03-21T08:43:24.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.010+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 msgr2=0x7f8f50069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.010+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 0x7f8f50069fe0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8f64006aa0 tx=0x7f8f64008040 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.010+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 msgr2=0x7f8f74154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.010+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00c430 tx=0x7f8f6c00c8f0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 shutdown_connections 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f7415e050 0x7f8f74154300 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8f50067b20 0x7f8f50069fe0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8f740ccf40 0x7f8f740cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 --2- 172.21.15.28:0/2316771750 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8f74099c20 0x7f8f741531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 >> 172.21.15.28:0/2316771750 conn(0x7f8f740b34f0 msgr2=0x7f8f740ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:24.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 shutdown_connections 2024-03-21T08:43:24.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.014+0000 7f8f79e00640 1 -- 172.21.15.28:0/2316771750 wait complete. 2024-03-21T08:43:24.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-21T08:43:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 Processor -- start 2024-03-21T08:43:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 -- start start 2024-03-21T08:43:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f12f40b43e0 0x7f12f4071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12f4071f60 0x7f12f40ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f40cd0e0 0x7f12f415e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40cd8d0 con 0x7f12f4071f60 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f415bfd0 con 0x7f12f40b43e0 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12fa200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f415b230 con 0x7f12f40cd0e0 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12f4071f60 0x7f12f40ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12f4071f60 0x7f12f40ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51030/0 (socket says 172.21.15.28:51030) 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 -- 172.21.15.28:0/3840322416 learned_addr learned my addr 172.21.15.28:0/3840322416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12f3400640 1 --2- 172.21.15.28:0/3840322416 >> v2:172.21.15.151:3300/0 conn(0x7f12f40b43e0 0x7f12f4071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12f3e00640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f40cd0e0 0x7f12f415e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 -- 172.21.15.28:0/3840322416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f40cd0e0 msgr2=0x7f12f415e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:24.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f40cd0e0 0x7f12f415e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 -- 172.21.15.28:0/3840322416 >> v2:172.21.15.151:3300/0 conn(0x7f12f40b43e0 msgr2=0x7f12f4071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:24.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3840322416 >> v2:172.21.15.151:3300/0 conn(0x7f12f40b43e0 0x7f12f4071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.182+0000 7f12ebe00640 1 -- 172.21.15.28:0/3840322416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f415b4f0 con 0x7f12f4071f60 2024-03-21T08:43:24.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f4071f60 0x7f12f40ccba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f12f406feb0 tx=0x7f12dc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce41a5bf36d0178 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:24.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12f2a00640 1 -- 172.21.15.28:0/3840322416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b0070 con 0x7f12f4071f60 2024-03-21T08:43:24.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12f2a00640 1 -- 172.21.15.28:0/3840322416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc0ab600 con 0x7f12f4071f60 2024-03-21T08:43:24.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12f2a00640 1 -- 172.21.15.28:0/3840322416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0ab9d0 con 0x7f12f4071f60 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 -- 172.21.15.28:0/3840322416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f4071f60 msgr2=0x7f12f40ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f4071f60 0x7f12f40ccba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f12f406feb0 tx=0x7f12dc01c260 comp rx=0 tx=0).stop 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 -- 172.21.15.28:0/3840322416 shutdown_connections 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f40cd0e0 0x7f12f415e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 --2- 172.21.15.28:0/3840322416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f4071f60 0x7f12f40ccba0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 --2- 172.21.15.28:0/3840322416 >> v2:172.21.15.151:3300/0 conn(0x7f12f40b43e0 0x7f12f4071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 -- 172.21.15.28:0/3840322416 >> 172.21.15.28:0/3840322416 conn(0x7f12f40674f0 msgr2=0x7f12f406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 -- 172.21.15.28:0/3840322416 shutdown_connections 2024-03-21T08:43:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.186+0000 7f12fa200640 1 -- 172.21.15.28:0/3840322416 wait complete. 2024-03-21T08:43:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 Processor -- start 2024-03-21T08:43:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- start start 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f4071f60 0x7f12f4149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12f40cd0e0 0x7f12f4147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40b4940 con 0x7f12f40b43e0 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40b47c0 con 0x7f12f40cd0e0 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40b2a60 con 0x7f12f4071f60 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51034/0 (socket says 172.21.15.28:51034) 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 -- 172.21.15.28:0/3070141088 learned_addr learned my addr 172.21.15.28:0/3070141088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12f3e00640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12f40cd0e0 0x7f12f4147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12f3400640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f4071f60 0x7f12f4149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 -- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f4071f60 msgr2=0x7f12f4149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f4071f60 0x7f12f4149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 -- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12f40cd0e0 msgr2=0x7f12f4147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12f40cd0e0 0x7f12f4147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 -- 172.21.15.28:0/3070141088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f4147890 con 0x7f12f40b43e0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12ebe00640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0ae040 tx=0x7f12dc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b9070 con 0x7f12f40b43e0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc0b4400 con 0x7f12f40b43e0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0b0040 con 0x7f12f40b43e0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f41652e0 con 0x7f12f40b43e0 2024-03-21T08:43:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f414d0c0 con 0x7f12f40b43e0 2024-03-21T08:43:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.190+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f4068110 con 0x7f12f40b43e0 2024-03-21T08:43:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.194+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f12dc0bc070 con 0x7f12f40b43e0 2024-03-21T08:43:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.194+0000 7f12eb400640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 0x7f12d0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.194+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5730+0+0 (secure 0 0 0) 0x7f12dc11efe0 con 0x7f12f40b43e0 2024-03-21T08:43:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.194+0000 7f12f3400640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 0x7f12d0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.194+0000 7f12f3400640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 0x7f12d0069fe0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f12e400ba30 tx=0x7f12e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.198+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f12dc016110 con 0x7f12f40b43e0 2024-03-21T08:43:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:24.358+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f12f415e950 con 0x7f12f40b43e0 2024-03-21T08:43:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.014+0000 7f12eb400640 1 -- 172.21.15.28:0/3070141088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v46) v1 ==== 160+0+0 (secure 0 0 0) 0x7f12dc0efa00 con 0x7f12f40b43e0 2024-03-21T08:43:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'data_new' removed 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 msgr2=0x7f12d0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 0x7f12d0069fe0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f12e400ba30 tx=0x7f12e4008040 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 msgr2=0x7f12f414c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0ae040 tx=0x7f12dc019000 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 shutdown_connections 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f12f40cd0e0 0x7f12f4147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12f40b43e0 0x7f12f414c2a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f12d0067b20 0x7f12d0069fe0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 --2- 172.21.15.28:0/3070141088 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f12f4071f60 0x7f12f4149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 >> 172.21.15.28:0/3070141088 conn(0x7f12f40674f0 msgr2=0x7f12f414b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 shutdown_connections 2024-03-21T08:43:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.018+0000 7f12fa200640 1 -- 172.21.15.28:0/3070141088 wait complete. 2024-03-21T08:43:25.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-21T08:43:25.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 Processor -- start 2024-03-21T08:43:25.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 -- start start 2024-03-21T08:43:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fefc00b4800 0x7fefc00ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefc0072190 0x7fefc00cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc00cd670 con 0x7fefc00b4420 2024-03-21T08:43:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc015bf90 con 0x7fefc0072190 2024-03-21T08:43:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefc4e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc015b1d0 con 0x7fefc00b4800 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefc0072190 0x7fefc00cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefc0072190 0x7fefc00cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51038/0 (socket says 172.21.15.28:51038) 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 -- 172.21.15.28:0/2195013113 learned_addr learned my addr 172.21.15.28:0/2195013113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbd400640 1 --2- 172.21.15.28:0/2195013113 >> v2:172.21.15.151:3300/0 conn(0x7fefc00b4800 0x7fefc00ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 -- 172.21.15.28:0/2195013113 >> v2:172.21.15.151:3300/0 conn(0x7fefc00b4800 msgr2=0x7fefc00ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 --2- 172.21.15.28:0/2195013113 >> v2:172.21.15.151:3300/0 conn(0x7fefc00b4800 0x7fefc00ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 -- 172.21.15.28:0/2195013113 >> v1:172.21.15.151:6789/0 conn(0x7fefc00b4420 legacy=0x7fefc0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.186+0000 7fefbe800640 1 -- 172.21.15.28:0/2195013113 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc015b4b0 con 0x7fefc0072190 2024-03-21T08:43:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefbe800640 1 --2- 172.21.15.28:0/2195013113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0072190 0x7fefc00cd130 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fefc00cc6f0 tx=0x7fefb801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347cf9c59ff0fcba server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefbca00640 1 -- 172.21.15.28:0/2195013113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80a5880 con 0x7fefc0072190 2024-03-21T08:43:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefbca00640 1 -- 172.21.15.28:0/2195013113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb80a5a20 con 0x7fefc0072190 2024-03-21T08:43:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefbca00640 1 -- 172.21.15.28:0/2195013113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80a5df0 con 0x7fefc0072190 2024-03-21T08:43:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 -- 172.21.15.28:0/2195013113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0072190 msgr2=0x7fefc00cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2195013113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0072190 0x7fefc00cd130 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fefc00cc6f0 tx=0x7fefb801c9c0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 -- 172.21.15.28:0/2195013113 shutdown_connections 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2195013113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0072190 0x7fefc00cd130 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2195013113 >> v2:172.21.15.151:3300/0 conn(0x7fefc00b4800 0x7fefc00ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 -- 172.21.15.28:0/2195013113 >> 172.21.15.28:0/2195013113 conn(0x7fefc00674f0 msgr2=0x7fefc006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 -- 172.21.15.28:0/2195013113 shutdown_connections 2024-03-21T08:43:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.190+0000 7fefc4e00640 1 -- 172.21.15.28:0/2195013113 wait complete. 2024-03-21T08:43:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 Processor -- start 2024-03-21T08:43:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- start start 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefc0072190 0x7fefc015d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fefc01452b0 0x7fefc015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b3460 con 0x7fefc00b4420 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b32e0 con 0x7fefc0072190 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b35e0 con 0x7fefc01452b0 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51044/0 (socket says 172.21.15.28:51044) 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 -- 172.21.15.28:0/2060461681 learned_addr learned my addr 172.21.15.28:0/2060461681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbde00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefc0072190 0x7fefc015d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbe800640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fefc01452b0 0x7fefc015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 -- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fefc01452b0 msgr2=0x7fefc015c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fefc01452b0 0x7fefc015c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 -- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefc0072190 msgr2=0x7fefc015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefc0072190 0x7fefc015d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 -- 172.21.15.28:0/2060461681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc014b200 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefbd400640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fefac00e840 tx=0x7fefac00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac017070 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac00ee90 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac00cb80 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc014b4e0 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc014bfc0 con 0x7fefc00b4420 2024-03-21T08:43:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.194+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc0073430 con 0x7fefc00b4420 2024-03-21T08:43:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.198+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fefac01a070 con 0x7fefc00b4420 2024-03-21T08:43:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.198+0000 7fefb6000640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 0x7fefa0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.198+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fefac085ec0 con 0x7fefc00b4420 2024-03-21T08:43:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.198+0000 7fefbde00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 0x7fefa0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.198+0000 7fefbde00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 0x7fefa0069fe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fefc01f7e00 tx=0x7fefb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.202+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fefac0563e0 con 0x7fefc00b4420 2024-03-21T08:43:25.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.366+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fefc006e990 con 0x7fefc00b4420 2024-03-21T08:43:25.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.366+0000 7fefb6000640 1 -- 172.21.15.28:0/2060461681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fefac05a480 con 0x7fefc00b4420 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 msgr2=0x7fefa0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 0x7fefa0069fe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fefc01f7e00 tx=0x7fefb0009040 comp rx=0 tx=0).stop 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 msgr2=0x7fefc0144990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fefac00e840 tx=0x7fefac00ec40 comp rx=0 tx=0).stop 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 shutdown_connections 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fefa0067b20 0x7fefa0069fe0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fefc01452b0 0x7fefc015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc00b4420 0x7fefc0144990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 --2- 172.21.15.28:0/2060461681 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fefc0072190 0x7fefc015d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 >> 172.21.15.28:0/2060461681 conn(0x7fefc00674f0 msgr2=0x7fefc0149190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 shutdown_connections 2024-03-21T08:43:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.370+0000 7fefc4e00640 1 -- 172.21.15.28:0/2060461681 wait complete. 2024-03-21T08:43:25.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-21T08:43:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 Processor -- start 2024-03-21T08:43:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 -- start start 2024-03-21T08:43:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e6415ba70 0x7f8e640b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6415ebf0 con 0x7f8e640ce490 2024-03-21T08:43:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6415eed0 con 0x7f8e6415e610 2024-03-21T08:43:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e68a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6415f1b0 con 0x7f8e6415ba70 2024-03-21T08:43:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e61e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e640ce490 0x7f8e640ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59570/0 (socket says 172.21.15.28:59570) 2024-03-21T08:43:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e61e00640 1 -- 172.21.15.28:0/3717619634 learned_addr learned my addr 172.21.15.28:0/3717619634 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e62800640 1 --2- 172.21.15.28:0/3717619634 >> v2:172.21.15.28:3300/0 conn(0x7f8e6415ba70 0x7f8e640b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e62800640 1 -- 172.21.15.28:0/3717619634 >> v1:172.21.15.151:6789/0 conn(0x7f8e6415e610 legacy=0x7f8e640b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e62800640 1 -- 172.21.15.28:0/3717619634 >> v1:172.21.15.28:6789/0 conn(0x7f8e640ce490 legacy=0x7f8e640ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.530+0000 7f8e62800640 1 -- 172.21.15.28:0/3717619634 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6415f490 con 0x7f8e6415ba70 2024-03-21T08:43:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e62800640 1 --2- 172.21.15.28:0/3717619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e640b4a90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009d90 tx=0x7f8e5c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2d9255f70fb41 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e61400640 1 -- 172.21.15.28:0/3717619634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0b2070 con 0x7f8e6415ba70 2024-03-21T08:43:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e61400640 1 -- 172.21.15.28:0/3717619634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c0a57a0 con 0x7f8e6415ba70 2024-03-21T08:43:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e61400640 1 -- 172.21.15.28:0/3717619634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0a5b70 con 0x7f8e6415ba70 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 -- 172.21.15.28:0/3717619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 msgr2=0x7f8e640b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 --2- 172.21.15.28:0/3717619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e640b4a90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009d90 tx=0x7f8e5c01c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 -- 172.21.15.28:0/3717619634 shutdown_connections 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 --2- 172.21.15.28:0/3717619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e640b4a90 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 -- 172.21.15.28:0/3717619634 >> 172.21.15.28:0/3717619634 conn(0x7f8e640cc230 msgr2=0x7f8e640cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 -- 172.21.15.28:0/3717619634 shutdown_connections 2024-03-21T08:43:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.534+0000 7f8e68a00640 1 -- 172.21.15.28:0/3717619634 wait complete. 2024-03-21T08:43:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 Processor -- start 2024-03-21T08:43:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- start start 2024-03-21T08:43:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e640ce490 0x7f8e6413e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e6415e610 0x7f8e6414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64168b40 con 0x7f8e6415ba70 2024-03-21T08:43:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641689c0 con 0x7f8e6415e610 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64168cc0 con 0x7f8e640ce490 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51074/0 (socket says 172.21.15.28:51074) 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 -- 172.21.15.28:0/28071199 learned_addr learned my addr 172.21.15.28:0/28071199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e61e00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e640ce490 0x7f8e6413e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e62800640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e6415e610 0x7f8e6414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 -- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e640ce490 msgr2=0x7f8e6413e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e640ce490 0x7f8e6413e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 -- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e6415e610 msgr2=0x7f8e6414c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e6415e610 0x7f8e6414c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 -- 172.21.15.28:0/28071199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6413e6f0 con 0x7f8e6415ba70 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5be00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c370 tx=0x7f8e4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0052d0 con 0x7f8e6415ba70 2024-03-21T08:43:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c005470 con 0x7f8e6415ba70 2024-03-21T08:43:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c007260 con 0x7f8e6415ba70 2024-03-21T08:43:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6414c0a0 con 0x7f8e6415ba70 2024-03-21T08:43:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.538+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e64154340 con 0x7f8e6415ba70 2024-03-21T08:43:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6404f410 con 0x7f8e6415ba70 2024-03-21T08:43:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f8e4c007400 con 0x7f8e6415ba70 2024-03-21T08:43:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e5a000640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 0x7f8e40069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f8e4c085110 con 0x7f8e6415ba70 2024-03-21T08:43:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e61e00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 0x7f8e40069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.542+0000 7f8e61e00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 0x7f8e40069fe0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8e54005890 tx=0x7f8e54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.546+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8e4c055630 con 0x7f8e6415ba70 2024-03-21T08:43:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:25.770+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7f8e64166790 con 0x7f8e6415ba70 2024-03-21T08:43:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e5a000640 1 -- 172.21.15.28:0/28071199 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v149) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8e4c0596d0 con 0x7f8e6415ba70 2024-03-21T08:43:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 msgr2=0x7f8e40069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 0x7f8e40069fe0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8e54005890 tx=0x7f8e54009040 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 msgr2=0x7f8e6414bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c370 tx=0x7f8e4c00c830 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 shutdown_connections 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f8e6415e610 0x7f8e6414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6415ba70 0x7f8e6414bb60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f8e40067b20 0x7f8e40069fe0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 --2- 172.21.15.28:0/28071199 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f8e640ce490 0x7f8e6413e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 >> 172.21.15.28:0/28071199 conn(0x7f8e640cc230 msgr2=0x7f8e64151310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 shutdown_connections 2024-03-21T08:43:26.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.018+0000 7f8e68a00640 1 -- 172.21.15.28:0/28071199 wait complete. 2024-03-21T08:43:26.045 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-21T08:43:26.045 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:43:26.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 Processor -- start 2024-03-21T08:43:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 -- start start 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd6c0ce490 0x7fdd6c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fdd6c0b4380 0x7fdd6c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c15bf00 con 0x7fdd6c15ba70 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd6c15ebf0 con 0x7fdd6c0ce490 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd71c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd6c15eed0 con 0x7fdd6c0b4380 2024-03-21T08:43:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd6c0ce490 0x7fdd6c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd6c0ce490 0x7fdd6c15e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51080/0 (socket says 172.21.15.28:51080) 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 -- 172.21.15.28:0/4060717469 learned_addr learned my addr 172.21.15.28:0/4060717469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 -- 172.21.15.28:0/4060717469 >> v2:172.21.15.151:3300/0 conn(0x7fdd6c0b4380 msgr2=0x7fdd6c0b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/4060717469 >> v2:172.21.15.151:3300/0 conn(0x7fdd6c0b4380 0x7fdd6c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 -- 172.21.15.28:0/4060717469 >> v1:172.21.15.28:6789/0 conn(0x7fdd6c15ba70 legacy=0x7fdd6c0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.190+0000 7fdd6aa00640 1 -- 172.21.15.28:0/4060717469 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c15f1b0 con 0x7fdd6c0ce490 2024-03-21T08:43:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/4060717469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c15e610 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0cf780 tx=0x7fdd5c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9081da1238606c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd6a000640 1 -- 172.21.15.28:0/4060717469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0b2070 con 0x7fdd6c0ce490 2024-03-21T08:43:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd6a000640 1 -- 172.21.15.28:0/4060717469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c0ad680 con 0x7fdd6c0ce490 2024-03-21T08:43:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd6a000640 1 -- 172.21.15.28:0/4060717469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0ada50 con 0x7fdd6c0ce490 2024-03-21T08:43:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 -- 172.21.15.28:0/4060717469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0ce490 msgr2=0x7fdd6c15e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 --2- 172.21.15.28:0/4060717469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c15e610 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0cf780 tx=0x7fdd5c01c490 comp rx=0 tx=0).stop 2024-03-21T08:43:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 -- 172.21.15.28:0/4060717469 shutdown_connections 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 --2- 172.21.15.28:0/4060717469 >> v2:172.21.15.151:3300/0 conn(0x7fdd6c0b4380 0x7fdd6c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 --2- 172.21.15.28:0/4060717469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c15e610 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 -- 172.21.15.28:0/4060717469 >> 172.21.15.28:0/4060717469 conn(0x7fdd6c0cc230 msgr2=0x7fdd6c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 -- 172.21.15.28:0/4060717469 shutdown_connections 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.194+0000 7fdd71c00640 1 -- 172.21.15.28:0/4060717469 wait complete. 2024-03-21T08:43:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 Processor -- start 2024-03-21T08:43:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- start start 2024-03-21T08:43:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd6c15ba70 0x7fdd6c1f8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c165b20 con 0x7fdd6c0b4380 2024-03-21T08:43:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c1659a0 con 0x7fdd6c0ce490 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c165ca0 con 0x7fdd6c15ba70 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51092/0 (socket says 172.21.15.28:51092) 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 -- 172.21.15.28:0/3091261880 learned_addr learned my addr 172.21.15.28:0/3091261880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 -- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd6c15ba70 msgr2=0x7fdd6c1f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd6c15ba70 0x7fdd6c1f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 -- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd6c0ce490 msgr2=0x7fdd6c140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 -- 172.21.15.28:0/3091261880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c145b50 con 0x7fdd6c0b4380 2024-03-21T08:43:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd6aa00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0190a0 tx=0x7fdd5c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0bc020 con 0x7fdd6c0b4380 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c0add80 con 0x7fdd6c0b4380 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0b2040 con 0x7fdd6c0b4380 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c146670 con 0x7fdd6c0b4380 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.198+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c1460b0 con 0x7fdd6c0b4380 2024-03-21T08:43:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c04f410 con 0x7fdd6c0b4380 2024-03-21T08:43:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fdd5c0bf020 con 0x7fdd6c0b4380 2024-03-21T08:43:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd62a00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 0x7fdd48069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fdd5c120070 con 0x7fdd6c0b4380 2024-03-21T08:43:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd63e00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 0x7fdd48069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.202+0000 7fdd63e00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 0x7fdd48069fe0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd54007fd0 tx=0x7fdd54007980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.206+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdd5c0c6030 con 0x7fdd6c0b4380 2024-03-21T08:43:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.374+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd6c0cbfa0 con 0x7fdd6c0b4380 2024-03-21T08:43:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.374+0000 7fdd62a00640 1 -- 172.21.15.28:0/3091261880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13305 (secure 0 0 0) 0x7fdd6c145b50 con 0x7fdd6c0b4380 2024-03-21T08:43:26.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:26.378 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":46,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:25.015033+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3938951152":"2024-03-22T08:43:21.006573+0000","172.21.15.28:6832/3938951152":"2024-03-22T08:43:21.006573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 msgr2=0x7fdd48069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 0x7fdd48069fe0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd54007fd0 tx=0x7fdd54007980 comp rx=0 tx=0).stop 2024-03-21T08:43:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 msgr2=0x7fdd6c145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0190a0 tx=0x7fdd5c01acd0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 shutdown_connections 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fdd48067b20 0x7fdd48069fe0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fdd6c15ba70 0x7fdd6c1f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fdd6c0ce490 0x7fdd6c140b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 --2- 172.21.15.28:0/3091261880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd6c0b4380 0x7fdd6c145610 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 >> 172.21.15.28:0/3091261880 conn(0x7fdd6c0cc230 msgr2=0x7fdd6c0b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 shutdown_connections 2024-03-21T08:43:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.378+0000 7fdd71c00640 1 -- 172.21.15.28:0/3091261880 wait complete. 2024-03-21T08:43:26.399 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:26.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 Processor -- start 2024-03-21T08:43:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 -- start start 2024-03-21T08:43:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb54006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb5400cde40 0x7fb540071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400723a0 con 0x7fb54015cbb0 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54015c720 con 0x7fb5400cde40 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb546e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ce5b0 con 0x7fb540071a80 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb544e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb54015cbb0 0x7fb5400cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59588/0 (socket says 172.21.15.28:59588) 2024-03-21T08:43:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb544e00640 1 -- 172.21.15.28:0/476181660 learned_addr learned my addr 172.21.15.28:0/476181660 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb537e00640 1 --2- 172.21.15.28:0/476181660 >> v2:172.21.15.151:3300/0 conn(0x7fb5400cde40 0x7fb540071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53fe00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb54006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484908076 0 0) 0x7fb5400723a0 con 0x7fb54015cbb0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524003530 con 0x7fb54015cbb0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282933455 0 0) 0x7fb524003530 con 0x7fb54015cbb0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 msgr2=0x7fb54006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb54006f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 >> v2:172.21.15.151:3300/0 conn(0x7fb5400cde40 msgr2=0x7fb540071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 --2- 172.21.15.28:0/476181660 >> v2:172.21.15.151:3300/0 conn(0x7fb5400cde40 0x7fb540071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400ce890 con 0x7fb54015cbb0 2024-03-21T08:43:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fb538001a00 con 0x7fb54015cbb0 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb537e00640 1 --2- 172.21.15.28:0/476181660 >> v2:172.21.15.151:3300/0 conn(0x7fb5400cde40 0x7fb540071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 >> v1:172.21.15.28:6789/0 conn(0x7fb54015cbb0 legacy=0x7fb5400cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5240039c0 0x7fb524003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb524004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015c720 con 0x7fb524004d50 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.542+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ce5b0 con 0x7fb5240039c0 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52400b9d0 con 0x7fb540071a80 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb537e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb524004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb53fe00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5240039c0 0x7fb524003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 -- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 msgr2=0x7fb524004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb540071a80 0x7fb524004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 -- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5240039c0 msgr2=0x7fb524003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5240039c0 0x7fb524003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 -- 172.21.15.28:0/476181660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb524003530 con 0x7fb524004d50 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb544e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5240051d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb53800a920 tx=0x7fb53801fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b47f5e835d881a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53800a4c0 con 0x7fb524004d50 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5380ad020 con 0x7fb524004d50 2024-03-21T08:43:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.546+0000 7fb53f400640 1 -- 172.21.15.28:0/476181660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5380a9730 con 0x7fb524004d50 2024-03-21T08:43:26.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- 172.21.15.28:0/476181660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 msgr2=0x7fb5240051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 --2- 172.21.15.28:0/476181660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5240051d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb53800a920 tx=0x7fb53801fba0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb544e00640 1 -- 172.21.15.28:0/476181660 reap_dead start 2024-03-21T08:43:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- 172.21.15.28:0/476181660 shutdown_connections 2024-03-21T08:43:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- 172.21.15.28:0/476181660 >> 172.21.15.28:0/476181660 conn(0x7fb5400674f0 msgr2=0x7fb54006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- 172.21.15.28:0/476181660 shutdown_connections 2024-03-21T08:43:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- 172.21.15.28:0/476181660 wait complete. 2024-03-21T08:43:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 Processor -- start 2024-03-21T08:43:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- start start 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5401f4330 0x7fb54013d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5400b02d0 0x7fb5400b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400ce2a0 con 0x7fb524004d50 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54015c0b0 con 0x7fb5400b02d0 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb546e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb540144b20 con 0x7fb5401f4330 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51118/0 (socket says 172.21.15.28:51118) 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 -- 172.21.15.28:0/2349419771 learned_addr learned my addr 172.21.15.28:0/2349419771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb537e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5401f4330 0x7fb54013d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb544e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5400b02d0 0x7fb5400b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 -- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5401f4330 msgr2=0x7fb54013d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5401f4330 0x7fb54013d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 -- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5400b02d0 msgr2=0x7fb5400b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5400b02d0 0x7fb5400b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.550+0000 7fb53fe00640 1 -- 172.21.15.28:0/2349419771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5401f48a0 con 0x7fb524004d50 2024-03-21T08:43:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53fe00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb5300119d0 tx=0x7fb530011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53000fb50 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53000fcf0 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53000fea0 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5401f4b80 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5401f5140 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb540073430 con 0x7fb524004d50 2024-03-21T08:43:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb530012040 con 0x7fb524004d50 2024-03-21T08:43:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 0x7fb51c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fb530086490 con 0x7fb524004d50 2024-03-21T08:43:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.554+0000 7fb537e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 0x7fb51c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.558+0000 7fb537e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 0x7fb51c06a010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb5400b16b0 tx=0x7fb528008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.558+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb530019030 con 0x7fb524004d50 2024-03-21T08:43:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5400cdc20 con 0x7fb524004d50 2024-03-21T08:43:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb53cc00640 1 -- 172.21.15.28:0/2349419771 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb530057030 con 0x7fb524004d50 2024-03-21T08:43:26.729 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 msgr2=0x7fb51c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 0x7fb51c06a010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb5400b16b0 tx=0x7fb528008040 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 msgr2=0x7fb5400c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb5300119d0 tx=0x7fb530011e90 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 shutdown_connections 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb5400b02d0 0x7fb5400b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb51c067b50 0x7fb51c06a010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb5401f4330 0x7fb54013d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 --2- 172.21.15.28:0/2349419771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb524004d50 0x7fb5400c5310 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 >> 172.21.15.28:0/2349419771 conn(0x7fb5400674f0 msgr2=0x7fb540071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 shutdown_connections 2024-03-21T08:43:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.726+0000 7fb546e00640 1 -- 172.21.15.28:0/2349419771 wait complete. 2024-03-21T08:43:26.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-21T08:43:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.886+0000 7f0f3fe00640 1 Processor -- start 2024-03-21T08:43:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.886+0000 7f0f3fe00640 1 -- start start 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f4009a140 0x7f0f400b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 0x7f0f400cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f400cd1d0 con 0x7f0f4015f220 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f400ccf90 con 0x7f0f4009a140 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3fe00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f400cd4b0 con 0x7f0f400c6400 2024-03-21T08:43:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f4009a140 0x7f0f400b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f4009a140 0x7f0f400b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51128/0 (socket says 172.21.15.28:51128) 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 -- 172.21.15.28:0/1680717401 learned_addr learned my addr 172.21.15.28:0/1680717401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3f400640 1 --2- 172.21.15.28:0/1680717401 >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 0x7f0f400cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 -- 172.21.15.28:0/1680717401 >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 msgr2=0x7f0f400cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 --2- 172.21.15.28:0/1680717401 >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 0x7f0f400cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 -- 172.21.15.28:0/1680717401 >> v1:172.21.15.28:6789/0 conn(0x7f0f4015f220 legacy=0x7f0f40099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3e000640 1 -- 172.21.15.28:0/1680717401 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434364101 0 0) 0x7f0f400cd1d0 con 0x7f0f4015f220 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f36000640 1 -- 172.21.15.28:0/1680717401 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f400cd790 con 0x7f0f4009a140 2024-03-21T08:43:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.890+0000 7f0f3f400640 1 --2- 172.21.15.28:0/1680717401 >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 0x7f0f400cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f36000640 1 --2- 172.21.15.28:0/1680717401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f400b2770 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0f400b4230 tx=0x7f0f2801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227019cd519c02e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3e000640 1 -- 172.21.15.28:0/1680717401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f280ab430 con 0x7f0f4009a140 2024-03-21T08:43:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3e000640 1 -- 172.21.15.28:0/1680717401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f280b0070 con 0x7f0f4009a140 2024-03-21T08:43:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3e000640 1 -- 172.21.15.28:0/1680717401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f280ab7f0 con 0x7f0f4009a140 2024-03-21T08:43:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- 172.21.15.28:0/1680717401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 msgr2=0x7f0f400b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/1680717401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f400b2770 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0f400b4230 tx=0x7f0f2801c520 comp rx=0 tx=0).stop 2024-03-21T08:43:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- 172.21.15.28:0/1680717401 shutdown_connections 2024-03-21T08:43:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/1680717401 >> v2:172.21.15.151:3300/0 conn(0x7f0f400c6400 0x7f0f400cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/1680717401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f400b2770 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- 172.21.15.28:0/1680717401 >> 172.21.15.28:0/1680717401 conn(0x7f0f400b3390 msgr2=0x7f0f400b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- 172.21.15.28:0/1680717401 shutdown_connections 2024-03-21T08:43:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- 172.21.15.28:0/1680717401 wait complete. 2024-03-21T08:43:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 Processor -- start 2024-03-21T08:43:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 -- start start 2024-03-21T08:43:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.894+0000 7f0f3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 0x7f0f4014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0f4015f220 0x7f0f400709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f400d01c0 con 0x7f0f4009a140 2024-03-21T08:43:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f400d0340 con 0x7f0f400c6400 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f400d04c0 con 0x7f0f4015f220 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51140/0 (socket says 172.21.15.28:51140) 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 -- 172.21.15.28:0/4272945537 learned_addr learned my addr 172.21.15.28:0/4272945537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f36000640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 0x7f0f4014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3f400640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0f4015f220 0x7f0f400709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 -- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0f4015f220 msgr2=0x7f0f400709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0f4015f220 0x7f0f400709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 -- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 msgr2=0x7f0f4014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 0x7f0f4014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 -- 172.21.15.28:0/4272945537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4014d140 con 0x7f0f4009a140 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f36000640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 0x7f0f4014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3ea00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000c7a0 tx=0x7f0f3000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30017070 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f30003db0 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30005270 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4014d420 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4014d9e0 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f40075270 con 0x7f0f4009a140 2024-03-21T08:43:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.898+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0f3001a070 con 0x7f0f4009a140 2024-03-21T08:43:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.902+0000 7f0f37e00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 0x7f0f1c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.902+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0f30085410 con 0x7f0f4009a140 2024-03-21T08:43:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.902+0000 7f0f36000640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 0x7f0f1c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.902+0000 7f0f36000640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 0x7f0f1c069fe0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f280af000 tx=0x7f0f280a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:26.902+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0f300558a0 con 0x7f0f4009a140 2024-03-21T08:43:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:27.126+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7f0f400cdfb0 con 0x7f0f4009a140 2024-03-21T08:43:28.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f37e00640 1 -- 172.21.15.28:0/4272945537 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v151) v1 ==== 137+0+0 (secure 0 0 0) 0x7f0f30059940 con 0x7f0f4009a140 2024-03-21T08:43:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 msgr2=0x7f0f1c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 0x7f0f1c069fe0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f280af000 tx=0x7f0f280a7000 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 msgr2=0x7f0f40155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000c7a0 tx=0x7f0f3000cc60 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 shutdown_connections 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0f1c067b20 0x7f0f1c069fe0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0f4015f220 0x7f0f400709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0f400c6400 0x7f0f4014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 --2- 172.21.15.28:0/4272945537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4009a140 0x7f0f40155880 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 >> 172.21.15.28:0/4272945537 conn(0x7f0f400b3390 msgr2=0x7f0f400d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 shutdown_connections 2024-03-21T08:43:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.026+0000 7f0f3fe00640 1 -- 172.21.15.28:0/4272945537 wait complete. 2024-03-21T08:43:28.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-21T08:43:28.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.198+0000 7f617b400640 1 Processor -- start 2024-03-21T08:43:28.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.198+0000 7f617b400640 1 -- start start 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f617b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 0x7f61740b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f617b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740b2d20 con 0x7f6174144690 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f617b400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61740b3000 con 0x7f6174071a80 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f617b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61740b2ad0 con 0x7f61741442b0 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f616be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6174144690 0x7f61740aebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59608/0 (socket says 172.21.15.28:59608) 2024-03-21T08:43:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f616be00640 1 -- 172.21.15.28:0/2920655538 learned_addr learned my addr 172.21.15.28:0/2920655538 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6178a00640 1 --2- 172.21.15.28:0/2920655538 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 0x7f61740b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490038843 0 0) 0x7f61740b2d20 con 0x7f6174144690 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158003530 con 0x7f6174144690 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150311402 0 0) 0x7f61740b3000 con 0x7f6174071a80 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61740b2d20 con 0x7f6174071a80 2024-03-21T08:43:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449160177 0 0) 0x7f6158003530 con 0x7f6174144690 2024-03-21T08:43:28.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6178a00640 1 -- 172.21.15.28:0/2920655538 >> v1:172.21.15.151:6789/0 conn(0x7f6174071a80 legacy=0x7f617406daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6178a00640 1 -- 172.21.15.28:0/2920655538 >> v1:172.21.15.28:6789/0 conn(0x7f6174144690 legacy=0x7f61740aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.202+0000 7f6178a00640 1 -- 172.21.15.28:0/2920655538 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740b32e0 con 0x7f61741442b0 2024-03-21T08:43:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f6178a00640 1 --2- 172.21.15.28:0/2920655538 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 0x7f61740b3dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61740b2c50 tx=0x7f616401c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=885ca121d13d0f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640b0070 con 0x7f61741442b0 2024-03-21T08:43:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640ab400 con 0x7f61741442b0 2024-03-21T08:43:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f6173e00640 1 -- 172.21.15.28:0/2920655538 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c001680 con 0x7f61741442b0 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- 172.21.15.28:0/2920655538 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 msgr2=0x7f61740b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 --2- 172.21.15.28:0/2920655538 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 0x7f61740b3dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f61740b2c50 tx=0x7f616401c3b0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- 172.21.15.28:0/2920655538 shutdown_connections 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 --2- 172.21.15.28:0/2920655538 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f61741442b0 0x7f61740b3dc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- 172.21.15.28:0/2920655538 >> 172.21.15.28:0/2920655538 conn(0x7f61740674f0 msgr2=0x7f617406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- 172.21.15.28:0/2920655538 shutdown_connections 2024-03-21T08:43:28.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- 172.21.15.28:0/2920655538 wait complete. 2024-03-21T08:43:28.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 Processor -- start 2024-03-21T08:43:28.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.206+0000 7f617b400640 1 -- start start 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6174071a80 0x7f61740b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61741442b0 0x7f6174168450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61740722a0 con 0x7f6174144690 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617416bb20 con 0x7f61741442b0 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61741427c0 con 0x7f6174071a80 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51152/0 (socket says 172.21.15.28:51152) 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 -- 172.21.15.28:0/547919483 learned_addr learned my addr 172.21.15.28:0/547919483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6178a00640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6174071a80 0x7f61740b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f616be00640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61741442b0 0x7f6174168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 -- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6174071a80 msgr2=0x7f61740b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6174071a80 0x7f61740b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 -- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61741442b0 msgr2=0x7f6174168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61741442b0 0x7f6174168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 -- 172.21.15.28:0/547919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174168990 con 0x7f6174144690 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6179400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f616c00c730 tx=0x7f616c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c003b80 con 0x7f6174144690 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c003d20 con 0x7f6174144690 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c005310 con 0x7f6174144690 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174141bb0 con 0x7f6174144690 2024-03-21T08:43:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174142170 con 0x7f6174144690 2024-03-21T08:43:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6174073430 con 0x7f6174144690 2024-03-21T08:43:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.210+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f616c0056c0 con 0x7f6174144690 2024-03-21T08:43:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.214+0000 7f6171600640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 0x7f615006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.214+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f616c085e90 con 0x7f6174144690 2024-03-21T08:43:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.214+0000 7f6178a00640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 0x7f615006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.214+0000 7f6178a00640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 0x7f615006a010 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f61640ad000 tx=0x7f61640a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.218+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f616c056970 con 0x7f6174144690 2024-03-21T08:43:28.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.382+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f617404f4f0 con 0x7f6174144690 2024-03-21T08:43:28.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f6171600640 1 -- 172.21.15.28:0/547919483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f616c087070 con 0x7f6174144690 2024-03-21T08:43:28.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 msgr2=0x7f615006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 0x7f615006a010 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f61640ad000 tx=0x7f61640a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 msgr2=0x7f617413bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f616c00c730 tx=0x7f616c00cbf0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 shutdown_connections 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6174144690 0x7f617413bd40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f61741442b0 0x7f6174168450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6150067b50 0x7f615006a010 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 --2- 172.21.15.28:0/547919483 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6174071a80 0x7f61740b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 >> 172.21.15.28:0/547919483 conn(0x7f61740674f0 msgr2=0x7f61740aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 shutdown_connections 2024-03-21T08:43:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.386+0000 7f617b400640 1 -- 172.21.15.28:0/547919483 wait complete. 2024-03-21T08:43:28.408 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-21T08:43:28.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:43:28.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:43:28.556 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-21T08:43:28.557 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:28.557 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:28.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:43:28.566 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:43:28.566 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:28.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:43:28.575 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:28.575 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:28.615 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:43:28.615 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:43:28.615 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:28.626 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:43:28.626 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-21T08:43:28.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.794+0000 7f6f98c00640 1 Processor -- start 2024-03-21T08:43:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.794+0000 7f6f98c00640 1 -- start start 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f98c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6f94099b80 0x7f6f9409a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f98c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f940cfcb0 0x7f6f940d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f98c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f940cc1d0 con 0x7f6f9415e0c0 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f98c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f940d1730 con 0x7f6f940cfcb0 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f98c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f940d1450 con 0x7f6f94099b80 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f91e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6f9415e0c0 0x7f6f9406cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59610/0 (socket says 172.21.15.28:59610) 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f91e00640 1 -- 172.21.15.28:0/3714428790 learned_addr learned my addr 172.21.15.28:0/3714428790 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f92800640 1 --2- 172.21.15.28:0/3714428790 >> v2:172.21.15.28:3300/0 conn(0x7f6f940cfcb0 0x7f6f940d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f8be00640 1 --2- 172.21.15.28:0/3714428790 >> v2:172.21.15.151:3300/0 conn(0x7f6f94099b80 0x7f6f9409a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f91400640 1 -- 172.21.15.28:0/3714428790 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135520639 0 0) 0x7f6f940cc1d0 con 0x7f6f9415e0c0 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f91400640 1 -- 172.21.15.28:0/3714428790 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78003530 con 0x7f6f9415e0c0 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f92800640 1 -- 172.21.15.28:0/3714428790 >> v2:172.21.15.151:3300/0 conn(0x7f6f94099b80 msgr2=0x7f6f9409a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f92800640 1 --2- 172.21.15.28:0/3714428790 >> v2:172.21.15.151:3300/0 conn(0x7f6f94099b80 0x7f6f9409a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f92800640 1 -- 172.21.15.28:0/3714428790 >> v1:172.21.15.28:6789/0 conn(0x7f6f9415e0c0 legacy=0x7f6f9406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.798+0000 7f6f92800640 1 -- 172.21.15.28:0/3714428790 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f940cc470 con 0x7f6f940cfcb0 2024-03-21T08:43:28.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f92800640 1 --2- 172.21.15.28:0/3714428790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f940cfcb0 0x7f6f940d0120 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6f94072ea0 tx=0x7f6f8c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8e923e0a0d0509 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f91400640 1 -- 172.21.15.28:0/3714428790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0b2070 con 0x7f6f940cfcb0 2024-03-21T08:43:28.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f91400640 1 -- 172.21.15.28:0/3714428790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0a5870 con 0x7f6f940cfcb0 2024-03-21T08:43:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f91400640 1 -- 172.21.15.28:0/3714428790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0a5c40 con 0x7f6f940cfcb0 2024-03-21T08:43:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- 172.21.15.28:0/3714428790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f940cfcb0 msgr2=0x7f6f940d0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3714428790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f940cfcb0 0x7f6f940d0120 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6f94072ea0 tx=0x7f6f8c01c930 comp rx=0 tx=0).stop 2024-03-21T08:43:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- 172.21.15.28:0/3714428790 shutdown_connections 2024-03-21T08:43:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3714428790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f940cfcb0 0x7f6f940d0120 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3714428790 >> v2:172.21.15.151:3300/0 conn(0x7f6f94099b80 0x7f6f9409a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- 172.21.15.28:0/3714428790 >> 172.21.15.28:0/3714428790 conn(0x7f6f940b34d0 msgr2=0x7f6f940b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- 172.21.15.28:0/3714428790 shutdown_connections 2024-03-21T08:43:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- 172.21.15.28:0/3714428790 wait complete. 2024-03-21T08:43:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 Processor -- start 2024-03-21T08:43:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- start start 2024-03-21T08:43:28.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 0x7f6f940c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f940cfcb0 0x7f6f9414eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.802+0000 7f6f98c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940d0e70 con 0x7f6f9415e0c0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f98c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940d0ff0 con 0x7f6f94099b80 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f98c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940d1170 con 0x7f6f940cfcb0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51174/0 (socket says 172.21.15.28:51174) 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 -- 172.21.15.28:0/3422261809 learned_addr learned my addr 172.21.15.28:0/3422261809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 -- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f940cfcb0 msgr2=0x7f6f9414eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f940cfcb0 0x7f6f9414eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 -- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 msgr2=0x7f6f940c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f91e00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 0x7f6f940c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 0x7f6f940c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 -- 172.21.15.28:0/3422261809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f940c88c0 con 0x7f6f9415e0c0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f91e00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 0x7f6f940c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f92800640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c019540 tx=0x7f6f8c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0b2050 con 0x7f6f9415e0c0 2024-03-21T08:43:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0be070 con 0x7f6f9415e0c0 2024-03-21T08:43:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0ad890 con 0x7f6f9415e0c0 2024-03-21T08:43:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f940c9a00 con 0x7f6f9415e0c0 2024-03-21T08:43:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f940c7900 con 0x7f6f9415e0c0 2024-03-21T08:43:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f940b4dd0 con 0x7f6f9415e0c0 2024-03-21T08:43:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.806+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6f8c0ada30 con 0x7f6f9415e0c0 2024-03-21T08:43:28.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.810+0000 7f6f8a000640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 0x7f6f7006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:28.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.810+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6f8c1207a0 con 0x7f6f9415e0c0 2024-03-21T08:43:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.810+0000 7f6f91e00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 0x7f6f7006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.810+0000 7f6f91e00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 0x7f6f7006a010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6f84005890 tx=0x7f6f84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:28.814+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6f940b4dd0 con 0x7f6f9415e0c0 2024-03-21T08:43:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f9404f410 con 0x7f6f9415e0c0 2024-03-21T08:43:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f8a000640 1 -- 172.21.15.28:0/3422261809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f6f8c0c7020 con 0x7f6f9415e0c0 2024-03-21T08:43:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:29.013 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5150,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3091019222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3091019222},{"type":"v1","addr":"172.21.15.151:6836","nonce":3091019222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5500,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/229985365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":229985365},{"type":"v1","addr":"172.21.15.151:6837","nonce":229985365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5506,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3597131299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3597131299},{"type":"v1","addr":"172.21.15.28:6835","nonce":3597131299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2895302314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2895302314},{"type":"v1","addr":"172.21.15.28:6834","nonce":2895302314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-03-21T08:43:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 21 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 msgr2=0x7f6f7006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 0x7f6f7006a010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6f84005890 tx=0x7f6f84009040 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 msgr2=0x7f6f940c9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c019540 tx=0x7f6f8c0196d0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 shutdown_connections 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f9415e0c0 0x7f6f940c9f10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6f70067b50 0x7f6f7006a010 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6f940cfcb0 0x7f6f9414eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 --2- 172.21.15.28:0/3422261809 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6f94099b80 0x7f6f940c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 >> 172.21.15.28:0/3422261809 conn(0x7f6f940b34d0 msgr2=0x7f6f9415bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 shutdown_connections 2024-03-21T08:43:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.010+0000 7f6f98c00640 1 -- 172.21.15.28:0/3422261809 wait complete. 2024-03-21T08:43:29.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-21T08:43:29.034 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-21T08:43:29.035 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-21T08:43:29.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:29.035 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-21T08:43:29.035 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-21T08:43:29.035 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-21T08:43:29.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:29.035 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-21T08:43:29.035 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-21T08:43:29.036 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-21T08:43:29.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:29.036 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-21T08:43:29.036 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-21T08:43:29.036 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-21T08:43:29.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:29.051 INFO:tasks.ceph.mds.d:Stopped 2024-03-21T08:43:29.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-21T08:43:29.052 INFO:tasks.ceph.mds.b:Stopped 2024-03-21T08:43:29.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-21T08:43:29.053 INFO:tasks.ceph.mds.a:Stopped 2024-03-21T08:43:29.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-21T08:43:29.054 INFO:tasks.ceph.mds.c:Stopped 2024-03-21T08:43:29.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-21T08:43:29.096 INFO:tasks.ceph.mds.d:Started 2024-03-21T08:43:29.097 DEBUG:teuthology.parallel:result is None 2024-03-21T08:43:29.097 INFO:tasks.ceph.mds.a:Started 2024-03-21T08:43:29.098 DEBUG:teuthology.parallel:result is None 2024-03-21T08:43:29.098 INFO:tasks.ceph.mds.b:Started 2024-03-21T08:43:29.098 DEBUG:teuthology.parallel:result is None 2024-03-21T08:43:29.099 INFO:tasks.ceph.mds.c:Started 2024-03-21T08:43:29.099 DEBUG:teuthology.parallel:result is None 2024-03-21T08:43:29.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-21T08:43:29.186 INFO:tasks.ceph.mds.b.smithi151.stdout:starting mds.b at 2024-03-21T08:43:29.187 INFO:tasks.ceph.mds.d.smithi151.stdout:starting mds.d at 2024-03-21T08:43:29.187 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-03-21T08:43:29.188 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-03-21T08:43:29.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 Processor -- start 2024-03-21T08:43:29.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 -- start start 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa5b80ccb70 0x7fa5b80cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80d0760 con 0x7fa5b806b170 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80cfec0 con 0x7fa5b815e020 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5be600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80ce7b0 con 0x7fa5b80ccb70 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5b806b170 0x7fa5b815ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59628/0 (socket says 172.21.15.28:59628) 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b7e00640 1 -- 172.21.15.28:0/788028453 learned_addr learned my addr 172.21.15.28:0/788028453 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b6a00640 1 --2- 172.21.15.28:0/788028453 >> v2:172.21.15.151:3300/0 conn(0x7fa5b80ccb70 0x7fa5b80cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b6000640 1 -- 172.21.15.28:0/788028453 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499749307 0 0) 0x7fa5b80d0760 con 0x7fa5b806b170 2024-03-21T08:43:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b6a00640 1 -- 172.21.15.28:0/788028453 >> v1:172.21.15.151:6789/0 conn(0x7fa5b815e020 legacy=0x7fa5b80cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b6a00640 1 -- 172.21.15.28:0/788028453 >> v1:172.21.15.28:6789/0 conn(0x7fa5b806b170 legacy=0x7fa5b815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.246+0000 7fa5b6a00640 1 -- 172.21.15.28:0/788028453 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80d01a0 con 0x7fa5b80ccb70 2024-03-21T08:43:29.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5b6a00640 1 --2- 172.21.15.28:0/788028453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b80ccb70 0x7fa5b80cd000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009770 tx=0x7fa5a401c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d00283016c7710 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:29.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5b6000640 1 -- 172.21.15.28:0/788028453 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40b0070 con 0x7fa5b80ccb70 2024-03-21T08:43:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5b6000640 1 -- 172.21.15.28:0/788028453 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a40ab600 con 0x7fa5b80ccb70 2024-03-21T08:43:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5b6000640 1 -- 172.21.15.28:0/788028453 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40ab9d0 con 0x7fa5b80ccb70 2024-03-21T08:43:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 -- 172.21.15.28:0/788028453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b80ccb70 msgr2=0x7fa5b80cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 --2- 172.21.15.28:0/788028453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b80ccb70 0x7fa5b80cd000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009770 tx=0x7fa5a401c2d0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 -- 172.21.15.28:0/788028453 shutdown_connections 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 --2- 172.21.15.28:0/788028453 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b80ccb70 0x7fa5b80cd000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 -- 172.21.15.28:0/788028453 >> 172.21.15.28:0/788028453 conn(0x7fa5b8099a30 msgr2=0x7fa5b8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 -- 172.21.15.28:0/788028453 shutdown_connections 2024-03-21T08:43:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.250+0000 7fa5be600640 1 -- 172.21.15.28:0/788028453 wait complete. 2024-03-21T08:43:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 Processor -- start 2024-03-21T08:43:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- start start 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b806b170 0x7fa5b814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 0x7fa5b814fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b815e790 con 0x7fa5b815e020 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b816c340 con 0x7fa5b806b170 2024-03-21T08:43:29.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b816c4c0 con 0x7fa5b80ccb70 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b6a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 0x7fa5b814fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51224/0 (socket says 172.21.15.28:51224) 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 -- 172.21.15.28:0/3375555197 learned_addr learned my addr 172.21.15.28:0/3375555197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7400640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b806b170 0x7fa5b814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 -- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 msgr2=0x7fa5b814fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 0x7fa5b814fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 -- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b806b170 msgr2=0x7fa5b814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b806b170 0x7fa5b814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 -- 172.21.15.28:0/3375555197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80ca5a0 con 0x7fa5b815e020 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b6a00640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 0x7fa5b814fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:29.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5b7e00640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c7a0 tx=0x7fa5b000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0018070 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b000a640 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b80cab60 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000aab0 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b80c48b0 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.254+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b8150590 con 0x7fa5b815e020 2024-03-21T08:43:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.258+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa5b000ac50 con 0x7fa5b815e020 2024-03-21T08:43:29.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.258+0000 7fa5af400640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 0x7fa598069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:29.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.258+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa5b0085450 con 0x7fa5b815e020 2024-03-21T08:43:29.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.258+0000 7fa5b7400640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 0x7fa598069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:29.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.258+0000 7fa5b7400640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 0x7fa598069fe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8007fd0 tx=0x7fa5a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.262+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa5b00558f0 con 0x7fa5b815e020 2024-03-21T08:43:29.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:29.434+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa5b80ce3e0 con 0x7fa5b815e020 2024-03-21T08:43:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.038+0000 7fa5af400640 1 -- 172.21.15.28:0/3375555197 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa5b0059990 con 0x7fa5b815e020 2024-03-21T08:43:30.042 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-03-21T08:43:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 msgr2=0x7fa598069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 0x7fa598069fe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8007fd0 tx=0x7fa5a8009040 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 msgr2=0x7fa5b8156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c7a0 tx=0x7fa5b000cc60 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 shutdown_connections 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b815e020 0x7fa5b8156f40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa598067b20 0x7fa598069fe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b80ccb70 0x7fa5b814fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 --2- 172.21.15.28:0/3375555197 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b806b170 0x7fa5b814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 >> 172.21.15.28:0/3375555197 conn(0x7fa5b8099a30 msgr2=0x7fa5b81536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 shutdown_connections 2024-03-21T08:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.042+0000 7fa5be600640 1 -- 172.21.15.28:0/3375555197 wait complete. 2024-03-21T08:43:30.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-21T08:43:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 Processor -- start 2024-03-21T08:43:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 -- start start 2024-03-21T08:43:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5ab015a2e0 0x7f5ab0072b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab00cee70 0x7f5ab006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab015a6c0 con 0x7f5ab006e620 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00ce0b0 con 0x7f5ab015a2e0 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5ab6000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00ce390 con 0x7f5ab00cee70 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aafe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ab006e620 0x7f5ab00acec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59636/0 (socket says 172.21.15.28:59636) 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aafe00640 1 -- 172.21.15.28:0/1515990194 learned_addr learned my addr 172.21.15.28:0/1515990194 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1515990194 >> v2:172.21.15.151:3300/0 conn(0x7f5ab015a2e0 0x7f5ab0072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aa7e00640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab00cee70 0x7f5ab006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aaea00640 1 -- 172.21.15.28:0/1515990194 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097859638 0 0) 0x7f5ab015a6c0 con 0x7f5ab006e620 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.214+0000 7f5aaea00640 1 -- 172.21.15.28:0/1515990194 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94003530 con 0x7f5ab006e620 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaf400640 1 -- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab00cee70 msgr2=0x7f5ab006e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab00cee70 0x7f5ab006e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaf400640 1 -- 172.21.15.28:0/1515990194 >> v1:172.21.15.28:6789/0 conn(0x7f5ab006e620 legacy=0x7f5ab00acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaf400640 1 -- 172.21.15.28:0/1515990194 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00ce670 con 0x7f5ab015a2e0 2024-03-21T08:43:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab015a2e0 0x7f5ab0072b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097c0 tx=0x7f5aa001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103339bde55b3d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaea00640 1 -- 172.21.15.28:0/1515990194 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00b0070 con 0x7f5ab015a2e0 2024-03-21T08:43:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaea00640 1 -- 172.21.15.28:0/1515990194 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa00ab600 con 0x7f5ab015a2e0 2024-03-21T08:43:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.218+0000 7f5aaea00640 1 -- 172.21.15.28:0/1515990194 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00ab9d0 con 0x7f5ab015a2e0 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab015a2e0 msgr2=0x7f5ab0072b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab015a2e0 0x7f5ab0072b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097c0 tx=0x7f5aa001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- 172.21.15.28:0/1515990194 shutdown_connections 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab00cee70 0x7f5ab006e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1515990194 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab015a2e0 0x7f5ab0072b90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- 172.21.15.28:0/1515990194 >> 172.21.15.28:0/1515990194 conn(0x7f5ab00d0050 msgr2=0x7f5ab00d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- 172.21.15.28:0/1515990194 shutdown_connections 2024-03-21T08:43:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- 172.21.15.28:0/1515990194 wait complete. 2024-03-21T08:43:30.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 Processor -- start 2024-03-21T08:43:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- start start 2024-03-21T08:43:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab006e620 0x7f5ab00c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab015a2e0 0x7f5ab00c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0073250 con 0x7f5ab00cee70 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00730d0 con 0x7f5ab006e620 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5ab6000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00ad5d0 con 0x7f5ab015a2e0 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51228/0 (socket says 172.21.15.28:51228) 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 -- 172.21.15.28:0/1010756738 learned_addr learned my addr 172.21.15.28:0/1010756738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab006e620 0x7f5ab00c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aafe00640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab015a2e0 0x7f5ab00c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 -- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab015a2e0 msgr2=0x7f5ab00c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab015a2e0 0x7f5ab00c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 -- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab006e620 msgr2=0x7f5ab00c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab006e620 0x7f5ab00c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 -- 172.21.15.28:0/1010756738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00b3220 con 0x7f5ab00cee70 2024-03-21T08:43:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.222+0000 7f5aa7e00640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e8e0 tx=0x7f5a9800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98016070 con 0x7f5ab00cee70 2024-03-21T08:43:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9800cc40 con 0x7f5ab00cee70 2024-03-21T08:43:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab00c4590 con 0x7f5ab00cee70 2024-03-21T08:43:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800b040 con 0x7f5ab00cee70 2024-03-21T08:43:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab00c4b50 con 0x7f5ab00cee70 2024-03-21T08:43:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab004f410 con 0x7f5ab00cee70 2024-03-21T08:43:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5a98019070 con 0x7f5ab00cee70 2024-03-21T08:43:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5a98085cc0 con 0x7f5ab00cee70 2024-03-21T08:43:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.226+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.230+0000 7f5aaf400640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00ad000 tx=0x7f5aa00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.230+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5a9808c080 con 0x7f5ab00cee70 2024-03-21T08:43:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.454+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ab006c6e0 con 0x7f5ab00cee70 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.454+0000 7f5aa7400640 1 -- 172.21.15.28:0/1010756738 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f5a980561a0 con 0x7f5ab00cee70 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-03-21T08:43:30.458 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtl+trHARAAVeUAY/NXkRXJdfWXcNEaCg==", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlQySxDRAA2TpyxtlBbuQn0qvoIOJIQQ==", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:43:30.459 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlDRkWCRAAilhyLYX1c0vULfdMLzWWAg==", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-03-21T08:43:30.460 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlnDLEFBAA6RwfJD0dSbvyuYmxKzSszA==", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBr8vtlK0S6NRAAKeHWfA4L/uoAhGbJeQ1p2w==", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.461 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBu8vtlGs2gOhAAFHS1xJ/R1i2EgFOJRLnTlw==", 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-03-21T08:43:30.462 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBy8vtl5TVhBBAAxqdX889ZTIy+UrQxJ5f2fg==", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB18vtlpyuwCRAAUZq5F4NJnBLa2+w5o/JKyA==", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB78vtlOZsrExAAH0sYZ+QeHoo4NMVzEkGg3Q==", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQB+8vtld4LJGRAAl/n0wJSkXXTQBqPgxXyEHQ==", 2024-03-21T08:43:30.464 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCB8vtlzRpuHxAAeRNXpyRn0AxBrPcyKN9WHw==", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCE8vtlicn7JBAAv8yEBq2UMrr+rKEVspGVRQ==", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw==", 2024-03-21T08:43:30.466 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtl4oOHIhAARXBoaT5pt4yfDvlIpcpweg==", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtlouglHxAAs+T0GLqKcfGvetFHJxp01Q==", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBq8vtljj+9KBAAA3rqu+R19oGWN/hi0PVTdQ==", 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.468 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtl+FA5IhAAgiEdsNSEbXexGkarmJyHJg==", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtlA+USORAA7ZqV/pA9y1+4XBehbbumog==", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBp8vtl+il8NRAAUlToWvhBAZuAqKX8f58AOg==", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-03-21T08:43:30.470 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-03-21T08:43:30.471 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-03-21T08:43:30.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:30.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-21T08:43:30.471 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 msgr2=0x7f5a8c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00ad000 tx=0x7f5aa00a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 msgr2=0x7f5ab00c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e8e0 tx=0x7f5a9800eda0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 shutdown_connections 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab015a2e0 0x7f5ab00c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab00cee70 0x7f5ab00c7120 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 --2- 172.21.15.28:0/1010756738 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab006e620 0x7f5ab00c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 >> 172.21.15.28:0/1010756738 conn(0x7f5ab00d0050 msgr2=0x7f5ab00acec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 shutdown_connections 2024-03-21T08:43:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.458+0000 7f5ab6000640 1 -- 172.21.15.28:0/1010756738 wait complete. 2024-03-21T08:43:30.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 Processor -- start 2024-03-21T08:43:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 -- start start 2024-03-21T08:43:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6b806eae0 0x7ff6b80cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b815ecb0 con 0x7ff6b80724b0 2024-03-21T08:43:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b80cd790 con 0x7ff6b806eae0 2024-03-21T08:43:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bfa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b815ef90 con 0x7ff6b8144280 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6b806eae0 0x7ff6b80cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6b806eae0 0x7ff6b80cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43262/0 (socket says 172.21.15.28:43262) 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 -- 172.21.15.28:0/2753954816 learned_addr learned my addr 172.21.15.28:0/2753954816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bd000640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 -- 172.21.15.28:0/2753954816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 msgr2=0x7ff6b815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 -- 172.21.15.28:0/2753954816 >> v1:172.21.15.151:6789/0 conn(0x7ff6b80724b0 legacy=0x7ff6b806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.618+0000 7ff6bda00640 1 -- 172.21.15.28:0/2753954816 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b815f270 con 0x7ff6b806eae0 2024-03-21T08:43:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bda00640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b80cdc20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009e50 tx=0x7ff6b001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9abcf4696d8335 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6b7400640 1 -- 172.21.15.28:0/2753954816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00a58f0 con 0x7ff6b806eae0 2024-03-21T08:43:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6b7400640 1 -- 172.21.15.28:0/2753954816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b00a5a90 con 0x7ff6b806eae0 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6b7400640 1 -- 172.21.15.28:0/2753954816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00a5e60 con 0x7ff6b806eae0 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 -- 172.21.15.28:0/2753954816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 msgr2=0x7ff6b80cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b80cdc20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009e50 tx=0x7ff6b001ca00 comp rx=0 tx=0).stop 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 -- 172.21.15.28:0/2753954816 shutdown_connections 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b80cdc20 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/2753954816 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 -- 172.21.15.28:0/2753954816 >> 172.21.15.28:0/2753954816 conn(0x7ff6b80cc230 msgr2=0x7ff6b80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 -- 172.21.15.28:0/2753954816 shutdown_connections 2024-03-21T08:43:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.622+0000 7ff6bfa00640 1 -- 172.21.15.28:0/2753954816 wait complete. 2024-03-21T08:43:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 Processor -- start 2024-03-21T08:43:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 -- start start 2024-03-21T08:43:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b80724b0 0x7ff6b8160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b80ceaf0 con 0x7ff6b806eae0 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b80ce970 con 0x7ff6b80724b0 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b80cec70 con 0x7ff6b8144280 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43270/0 (socket says 172.21.15.28:43270) 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 -- 172.21.15.28:0/615791464 learned_addr learned my addr 172.21.15.28:0/615791464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6b7e00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b80724b0 0x7ff6b8160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bda00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 -- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 msgr2=0x7ff6b815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 -- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b80724b0 msgr2=0x7ff6b8160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b80724b0 0x7ff6b8160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 -- 172.21.15.28:0/615791464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b813cfb0 con 0x7ff6b806eae0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bd000640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00d5f0 tx=0x7ff6ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0125f0 con 0x7ff6b806eae0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac019070 con 0x7ff6b806eae0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b813d290 con 0x7ff6b806eae0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.626+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0129d0 con 0x7ff6b806eae0 2024-03-21T08:43:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b813d850 con 0x7ff6b806eae0 2024-03-21T08:43:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b804f410 con 0x7ff6b806eae0 2024-03-21T08:43:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff6ac012b70 con 0x7ff6b806eae0 2024-03-21T08:43:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6b4c00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 0x7ff69806a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff6ac085ff0 con 0x7ff6b806eae0 2024-03-21T08:43:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6b7e00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 0x7ff69806a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.630+0000 7ff6b7e00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 0x7ff69806a030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff6b81fa250 tx=0x7ff6a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.634+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff6ac056640 con 0x7ff6b806eae0 2024-03-21T08:43:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.798+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6b80720c0 con 0x7ff6b806eae0 2024-03-21T08:43:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6b4c00640 1 -- 172.21.15.28:0/615791464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13176 (secure 0 0 0) 0x7ff6ac05a6e0 con 0x7ff6b806eae0 2024-03-21T08:43:30.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:30.804 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":50,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:30.051769+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 msgr2=0x7ff69806a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 0x7ff69806a030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff6b81fa250 tx=0x7ff6a4009040 comp rx=0 tx=0).stop 2024-03-21T08:43:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 msgr2=0x7ff6b813f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00d5f0 tx=0x7ff6ac00dab0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 shutdown_connections 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff698067b70 0x7ff69806a030 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff6b8144280 0x7ff6b815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff6b80724b0 0x7ff6b8160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 --2- 172.21.15.28:0/615791464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6b806eae0 0x7ff6b813f4c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 >> 172.21.15.28:0/615791464 conn(0x7ff6b80cc230 msgr2=0x7ff6b8167b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 shutdown_connections 2024-03-21T08:43:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.802+0000 7ff6bfa00640 1 -- 172.21.15.28:0/615791464 wait complete. 2024-03-21T08:43:30.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:30.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 Processor -- start 2024-03-21T08:43:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 -- start start 2024-03-21T08:43:30.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 0x7f7aec0b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec15e5d0 con 0x7f7aec099bf0 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aec0cfbb0 con 0x7f7aec06d850 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7af2e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0d0170 con 0x7f7aec15e1f0 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7ae3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7aec099bf0 0x7f7aec0ccb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38590/0 (socket says 172.21.15.28:38590) 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7ae3400640 1 -- 172.21.15.28:0/273040289 learned_addr learned my addr 172.21.15.28:0/273040289 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aebe00640 1 --2- 172.21.15.28:0/273040289 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 0x7f7aec0b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901114432 0 0) 0x7f7aec15e5d0 con 0x7f7aec099bf0 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad0003530 con 0x7f7aec099bf0 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163233984 0 0) 0x7f7aec0cfbb0 con 0x7f7aec06d850 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aec15e5d0 con 0x7f7aec06d850 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686834384 0 0) 0x7f7ad0003530 con 0x7f7aec099bf0 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aebe00640 1 -- 172.21.15.28:0/273040289 >> v1:172.21.15.151:6789/0 conn(0x7f7aec06d850 legacy=0x7f7aec0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aebe00640 1 -- 172.21.15.28:0/273040289 >> v1:172.21.15.28:6789/0 conn(0x7f7aec099bf0 legacy=0x7f7aec0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.966+0000 7f7aebe00640 1 -- 172.21.15.28:0/273040289 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0cfe90 con 0x7f7aec15e1f0 2024-03-21T08:43:30.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7aebe00640 1 --2- 172.21.15.28:0/273040289 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 0x7f7aec0b0b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009820 tx=0x7f7adc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42666ca59b35630c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0b0070 con 0x7f7aec15e1f0 2024-03-21T08:43:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc0ab400 con 0x7f7aec15e1f0 2024-03-21T08:43:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7aeb400640 1 -- 172.21.15.28:0/273040289 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4001680 con 0x7f7aec15e1f0 2024-03-21T08:43:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 -- 172.21.15.28:0/273040289 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 msgr2=0x7f7aec0b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 --2- 172.21.15.28:0/273040289 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 0x7f7aec0b0b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009820 tx=0x7f7adc01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 -- 172.21.15.28:0/273040289 shutdown_connections 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 --2- 172.21.15.28:0/273040289 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec15e1f0 0x7f7aec0b0b20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 -- 172.21.15.28:0/273040289 >> 172.21.15.28:0/273040289 conn(0x7f7aec0b3600 msgr2=0x7f7aec0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 -- 172.21.15.28:0/273040289 shutdown_connections 2024-03-21T08:43:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.970+0000 7f7af2e00640 1 -- 172.21.15.28:0/273040289 wait complete. 2024-03-21T08:43:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 Processor -- start 2024-03-21T08:43:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- start start 2024-03-21T08:43:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec06d850 0x7f7aec14c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 0x7f7aec0c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0d0c60 con 0x7f7aec15e1f0 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec0d0ae0 con 0x7f7aec099bf0 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec16c570 con 0x7f7aec06d850 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43284/0 (socket says 172.21.15.28:43284) 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 -- 172.21.15.28:0/4216789177 learned_addr learned my addr 172.21.15.28:0/4216789177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7aebe00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec06d850 0x7f7aec14c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7ae3400640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 0x7f7aec0c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 -- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec06d850 msgr2=0x7f7aec14c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec06d850 0x7f7aec14c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 -- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 msgr2=0x7f7aec0c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 0x7f7aec0c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 -- 172.21.15.28:0/4216789177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec14c960 con 0x7f7aec15e1f0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7ae3400640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 0x7f7aec0c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af0e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c790 tx=0x7f7ae400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4003b80 con 0x7f7aec15e1f0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae4003d20 con 0x7f7aec15e1f0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400aa50 con 0x7f7aec15e1f0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec154950 con 0x7f7aec15e1f0 2024-03-21T08:43:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.974+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec154f10 con 0x7f7aec15e1f0 2024-03-21T08:43:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.978+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec09fac0 con 0x7f7aec15e1f0 2024-03-21T08:43:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.978+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f7ae400abf0 con 0x7f7aec15e1f0 2024-03-21T08:43:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.978+0000 7f7ae8c00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 0x7f7ac8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.978+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7ae4085c80 con 0x7f7aec15e1f0 2024-03-21T08:43:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.978+0000 7f7aebe00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 0x7f7ac8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.982+0000 7f7aebe00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 0x7f7ac8069fe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0ad000 tx=0x7f7adc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:30.982+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ae40567b0 con 0x7f7aec15e1f0 2024-03-21T08:43:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7aec070f00 con 0x7f7aec15e1f0 2024-03-21T08:43:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7ae8c00640 1 -- 172.21.15.28:0/4216789177 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ae40560b0 con 0x7f7aec15e1f0 2024-03-21T08:43:31.153 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 msgr2=0x7f7ac8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 0x7f7ac8069fe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0ad000 tx=0x7f7adc0a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 msgr2=0x7f7aec15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c790 tx=0x7f7ae400cc50 comp rx=0 tx=0).stop 2024-03-21T08:43:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 shutdown_connections 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec15e1f0 0x7f7aec15a280 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7aec099bf0 0x7f7aec0c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7ac8067b20 0x7f7ac8069fe0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 --2- 172.21.15.28:0/4216789177 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f7aec06d850 0x7f7aec14c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 >> 172.21.15.28:0/4216789177 conn(0x7f7aec0b3600 msgr2=0x7f7aec0b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 shutdown_connections 2024-03-21T08:43:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.150+0000 7f7af2e00640 1 -- 172.21.15.28:0/4216789177 wait complete. 2024-03-21T08:43:31.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-21T08:43:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.310+0000 7f005f200640 1 Processor -- start 2024-03-21T08:43:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.310+0000 7f005f200640 1 -- start start 2024-03-21T08:43:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005f200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f00580d0880 0x7f005815f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f005806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005f200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00580cfac0 con 0x7f0058099b80 2024-03-21T08:43:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005f200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00580cfda0 con 0x7f00580d0880 2024-03-21T08:43:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580d0080 con 0x7f00580b0b40 2024-03-21T08:43:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f0057e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f0058099b80 0x7f0058099f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:55938/0 (socket says 172.21.15.28:55938) 2024-03-21T08:43:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f0057e00640 1 -- 172.21.15.28:0/2973348863 learned_addr learned my addr 172.21.15.28:0/2973348863 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f004fe00640 1 --2- 172.21.15.28:0/2973348863 >> v2:172.21.15.151:3300/0 conn(0x7f00580d0880 0x7f005815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005d200640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f005806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f0057400640 1 -- 172.21.15.28:0/2973348863 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164020237 0 0) 0x7f00580cfac0 con 0x7f0058099b80 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f0057400640 1 -- 172.21.15.28:0/2973348863 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c003530 con 0x7f0058099b80 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f004fe00640 1 -- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 msgr2=0x7f005806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f004fe00640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f005806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f004fe00640 1 -- 172.21.15.28:0/2973348863 >> v1:172.21.15.151:6789/0 conn(0x7f0058099b80 legacy=0x7f0058099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f004fe00640 1 -- 172.21.15.28:0/2973348863 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580d0360 con 0x7f00580d0880 2024-03-21T08:43:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.314+0000 7f005d200640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f005806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f004fe00640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815f220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f00400097c0 tx=0x7f004001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524245b9b0d76f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f0057400640 1 -- 172.21.15.28:0/2973348863 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00400b1070 con 0x7f00580d0880 2024-03-21T08:43:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f0057400640 1 -- 172.21.15.28:0/2973348863 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00400ac620 con 0x7f00580d0880 2024-03-21T08:43:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f0057400640 1 -- 172.21.15.28:0/2973348863 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00400ac9f0 con 0x7f00580d0880 2024-03-21T08:43:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 msgr2=0x7f005815f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815f220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f00400097c0 tx=0x7f004001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- 172.21.15.28:0/2973348863 shutdown_connections 2024-03-21T08:43:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f005806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- 172.21.15.28:0/2973348863 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815f220 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- 172.21.15.28:0/2973348863 >> 172.21.15.28:0/2973348863 conn(0x7f00580b3510 msgr2=0x7f00580b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- 172.21.15.28:0/2973348863 shutdown_connections 2024-03-21T08:43:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- 172.21.15.28:0/2973348863 wait complete. 2024-03-21T08:43:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 Processor -- start 2024-03-21T08:43:31.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- start start 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f00581566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005806d860 con 0x7f0058099b80 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580b2d40 con 0x7f00580d0880 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.318+0000 7f005f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005816b050 con 0x7f00580b0b40 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43296/0 (socket says 172.21.15.28:43296) 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 -- 172.21.15.28:0/1313435050 learned_addr learned my addr 172.21.15.28:0/1313435050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f005d200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f004fe00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f00581566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 -- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 msgr2=0x7f00581566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f00581566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 -- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 msgr2=0x7f005815c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 -- 172.21.15.28:0/1313435050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0058155600 con 0x7f0058099b80 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f004fe00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f00581566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0057e00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f004800c3c0 tx=0x7f004800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0048017070 con 0x7f0058099b80 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0048005db0 con 0x7f0058099b80 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00480072f0 con 0x7f0058099b80 2024-03-21T08:43:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00581558e0 con 0x7f0058099b80 2024-03-21T08:43:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0058155ea0 con 0x7f0058099b80 2024-03-21T08:43:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00581562e0 con 0x7f0058099b80 2024-03-21T08:43:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.322+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f004801a070 con 0x7f0058099b80 2024-03-21T08:43:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.326+0000 7f0054c00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 0x7f0034069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.326+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0048085220 con 0x7f0058099b80 2024-03-21T08:43:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.326+0000 7f004fe00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 0x7f0034069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.326+0000 7f004fe00640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 0x7f0034069fe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00400ae000 tx=0x7f00400a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.330+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0048055de0 con 0x7f0058099b80 2024-03-21T08:43:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f005804f410 con 0x7f0058099b80 2024-03-21T08:43:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f0054c00640 1 -- 172.21.15.28:0/1313435050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+65 (secure 0 0 0) 0x7f00480556e0 con 0x7f0058099b80 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:31.496 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 msgr2=0x7f0034069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 0x7f0034069fe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00400ae000 tx=0x7f00400a7000 comp rx=0 tx=0).stop 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 msgr2=0x7f005815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f004800c3c0 tx=0x7f004800c7c0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 shutdown_connections 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f00580d0880 0x7f005815c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0034067b20 0x7f0034069fe0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f00580b0b40 0x7f00581566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 --2- 172.21.15.28:0/1313435050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0058099b80 0x7f005815e120 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 >> 172.21.15.28:0/1313435050 conn(0x7f00580b3510 msgr2=0x7f00580ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 shutdown_connections 2024-03-21T08:43:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.494+0000 7f005f200640 1 -- 172.21.15.28:0/1313435050 wait complete. 2024-03-21T08:43:31.517 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-21T08:43:31.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-21T08:43:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.658+0000 7f959ae00640 1 Processor -- start 2024-03-21T08:43:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.658+0000 7f959ae00640 1 -- start start 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959ae00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9594142230 0x7f959415d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 0x7f95940cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959415d6e0 con 0x7f9594071c50 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959ae00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9594072790 con 0x7f9594142230 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95940721d0 con 0x7f95940717f0 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959a400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9594071c50 0x7f959406e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38602/0 (socket says 172.21.15.28:38602) 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f959a400640 1 -- 172.21.15.28:0/1472449126 learned_addr learned my addr 172.21.15.28:0/1472449126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599000640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 0x7f95940cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599a00640 1 --2- 172.21.15.28:0/1472449126 >> v2:172.21.15.151:3300/0 conn(0x7f9594142230 0x7f959415d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9593e00640 1 -- 172.21.15.28:0/1472449126 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344507098 0 0) 0x7f959415d6e0 con 0x7f9594071c50 2024-03-21T08:43:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9593e00640 1 -- 172.21.15.28:0/1472449126 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c003530 con 0x7f9594071c50 2024-03-21T08:43:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599a00640 1 -- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 msgr2=0x7f95940cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599a00640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 0x7f95940cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599a00640 1 -- 172.21.15.28:0/1472449126 >> v1:172.21.15.28:6789/0 conn(0x7f9594071c50 legacy=0x7f959406e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599a00640 1 -- 172.21.15.28:0/1472449126 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940724b0 con 0x7f9594142230 2024-03-21T08:43:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.662+0000 7f9599000640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 0x7f95940cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f9599a00640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594142230 0x7f959415d1a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f95880097c0 tx=0x7f958801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27cfed42e8b7d44 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f9593e00640 1 -- 172.21.15.28:0/1472449126 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880b0070 con 0x7f9594142230 2024-03-21T08:43:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f9593e00640 1 -- 172.21.15.28:0/1472449126 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95880ab600 con 0x7f9594142230 2024-03-21T08:43:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f9593e00640 1 -- 172.21.15.28:0/1472449126 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880ab9d0 con 0x7f9594142230 2024-03-21T08:43:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594142230 msgr2=0x7f959415d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594142230 0x7f959415d1a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f95880097c0 tx=0x7f958801c380 comp rx=0 tx=0).stop 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- 172.21.15.28:0/1472449126 shutdown_connections 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f95940717f0 0x7f95940cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 --2- 172.21.15.28:0/1472449126 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594142230 0x7f959415d1a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- 172.21.15.28:0/1472449126 >> 172.21.15.28:0/1472449126 conn(0x7f9594067630 msgr2=0x7f959406b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- 172.21.15.28:0/1472449126 shutdown_connections 2024-03-21T08:43:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- 172.21.15.28:0/1472449126 wait complete. 2024-03-21T08:43:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 Processor -- start 2024-03-21T08:43:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.666+0000 7f959ae00640 1 -- start start 2024-03-21T08:43:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9594071c50 0x7f95940b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 0x7f9594156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9594169230 con 0x7f9594142230 2024-03-21T08:43:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959415f9e0 con 0x7f9594155540 2024-03-21T08:43:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95941693b0 con 0x7f9594071c50 2024-03-21T08:43:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43312/0 (socket says 172.21.15.28:43312) 2024-03-21T08:43:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 -- 172.21.15.28:0/50355477 learned_addr learned my addr 172.21.15.28:0/50355477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 -- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9594071c50 msgr2=0x7f95940b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599a00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9594071c50 0x7f95940b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9594071c50 0x7f95940b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959a400640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 0x7f9594156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 -- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 msgr2=0x7f9594156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 0x7f9594156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 -- 172.21.15.28:0/50355477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959415a2b0 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959a400640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 0x7f9594156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9599000640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f958000e980 tx=0x7f958000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580017070 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000ca00 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000cdd0 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959415a590 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959415ab50 con 0x7f9594142230 2024-03-21T08:43:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959404f410 con 0x7f9594142230 2024-03-21T08:43:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.670+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f958001a070 con 0x7f9594142230 2024-03-21T08:43:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.674+0000 7f9591600640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 0x7f9574069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.674+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9580085dc0 con 0x7f9594142230 2024-03-21T08:43:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.674+0000 7f9599a00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 0x7f9574069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.674+0000 7f9599a00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 0x7f9574069fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f95880ad000 tx=0x7f9588008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.678+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9580056860 con 0x7f9594142230 2024-03-21T08:43:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:31.838+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f95940cc950 con 0x7f9594142230 2024-03-21T08:43:32.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.042+0000 7f9591600640 1 -- 172.21.15.28:0/50355477 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v51) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9580056160 con 0x7f9594142230 2024-03-21T08:43:32.045 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-03-21T08:43:32.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 msgr2=0x7f9574069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 0x7f9574069fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f95880ad000 tx=0x7f9588008040 comp rx=0 tx=0).stop 2024-03-21T08:43:32.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 msgr2=0x7f95940b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f958000e980 tx=0x7f958000ee40 comp rx=0 tx=0).stop 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 shutdown_connections 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9594155540 0x7f9594156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9594142230 0x7f95940b3e10 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9574067b20 0x7f9574069fe0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 --2- 172.21.15.28:0/50355477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9594071c50 0x7f95940b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 >> 172.21.15.28:0/50355477 conn(0x7f95940717f0 msgr2=0x7f95941661a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 shutdown_connections 2024-03-21T08:43:32.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.046+0000 7f959ae00640 1 -- 172.21.15.28:0/50355477 wait complete. 2024-03-21T08:43:32.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-21T08:43:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 Processor -- start 2024-03-21T08:43:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 -- start start 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcdf8072190 0x7fcdf80cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf815bf90 con 0x7fcdf80b4420 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf815b1d0 con 0x7fcdf80b4800 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdff000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf815b4b0 con 0x7fcdf8072190 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdf7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcdf80b4420 0x7fcdf8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38610/0 (socket says 172.21.15.28:38610) 2024-03-21T08:43:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdf7e00640 1 -- 172.21.15.28:0/1693566906 learned_addr learned my addr 172.21.15.28:0/1693566906 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:32.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdfd000640 1 --2- 172.21.15.28:0/1693566906 >> v2:172.21.15.28:3300/0 conn(0x7fcdf8072190 0x7fcdf80cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:32.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdf7400640 1 -- 172.21.15.28:0/1693566906 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206722249 0 0) 0x7fcdf815bf90 con 0x7fcdf80b4420 2024-03-21T08:43:32.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdf7400640 1 -- 172.21.15.28:0/1693566906 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcddc005740 con 0x7fcdf80b4420 2024-03-21T08:43:32.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdfd000640 1 -- 172.21.15.28:0/1693566906 >> v1:172.21.15.151:6789/0 conn(0x7fcdf80b4800 legacy=0x7fcdf80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdfd000640 1 -- 172.21.15.28:0/1693566906 >> v1:172.21.15.28:6789/0 conn(0x7fcdf80b4420 legacy=0x7fcdf8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.218+0000 7fcdfd000640 1 -- 172.21.15.28:0/1693566906 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf815b790 con 0x7fcdf8072190 2024-03-21T08:43:32.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdfd000640 1 --2- 172.21.15.28:0/1693566906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf80cd300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009d90 tx=0x7fcdf001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1564ec2ed408aa2f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:32.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdf7400640 1 -- 172.21.15.28:0/1693566906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00b2070 con 0x7fcdf8072190 2024-03-21T08:43:32.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdf7400640 1 -- 172.21.15.28:0/1693566906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf00a58a0 con 0x7fcdf8072190 2024-03-21T08:43:32.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdf7400640 1 -- 172.21.15.28:0/1693566906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00a5c70 con 0x7fcdf8072190 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 -- 172.21.15.28:0/1693566906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 msgr2=0x7fcdf80cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 --2- 172.21.15.28:0/1693566906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf80cd300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009d90 tx=0x7fcdf001c940 comp rx=0 tx=0).stop 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 -- 172.21.15.28:0/1693566906 shutdown_connections 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 --2- 172.21.15.28:0/1693566906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf80cd300 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 -- 172.21.15.28:0/1693566906 >> 172.21.15.28:0/1693566906 conn(0x7fcdf80674f0 msgr2=0x7fcdf806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 -- 172.21.15.28:0/1693566906 shutdown_connections 2024-03-21T08:43:32.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.222+0000 7fcdff000640 1 -- 172.21.15.28:0/1693566906 wait complete. 2024-03-21T08:43:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 Processor -- start 2024-03-21T08:43:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- start start 2024-03-21T08:43:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 0x7fcdf81441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 0x7fcdf8145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf806e430 con 0x7fcdf8072190 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf806e2b0 con 0x7fcdf80b4420 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf806e5b0 con 0x7fcdf8153860 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43330/0 (socket says 172.21.15.28:43330) 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 -- 172.21.15.28:0/1472511176 learned_addr learned my addr 172.21.15.28:0/1472511176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdfd000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 0x7fcdf8145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdefe00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 0x7fcdf81441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 -- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 msgr2=0x7fcdf8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 0x7fcdf8145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 -- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 msgr2=0x7fcdf81441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 0x7fcdf81441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 -- 172.21.15.28:0/1472511176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80b4f00 con 0x7fcdf8072190 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdfd000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 0x7fcdf8145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdefe00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 0x7fcdf81441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf7e00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c500 tx=0x7fcde800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8017070 con 0x7fcdf8072190 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde8005cf0 con 0x7fcdf8072190 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde80072f0 con 0x7fcdf8072190 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf80ce1b0 con 0x7fcdf8072190 2024-03-21T08:43:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf8144d00 con 0x7fcdf8072190 2024-03-21T08:43:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.226+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf8073430 con 0x7fcdf8072190 2024-03-21T08:43:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.230+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcde801a070 con 0x7fcdf8072190 2024-03-21T08:43:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.230+0000 7fcdf4c00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 0x7fcdd4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.230+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fcde8085260 con 0x7fcdf8072190 2024-03-21T08:43:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.230+0000 7fcdefe00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 0x7fcdd4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.230+0000 7fcdefe00640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 0x7fcdd4069fe0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8144180 tx=0x7fcde0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:32.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.234+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcde8055660 con 0x7fcdf8072190 2024-03-21T08:43:32.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:32.398+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcdf804f4f0 con 0x7fcdf8072190 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdf4c00640 1 -- 172.21.15.28:0/1472511176 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v52) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcde8059700 con 0x7fcdf8072190 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 msgr2=0x7fcdd4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 0x7fcdd4069fe0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8144180 tx=0x7fcde0009040 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 msgr2=0x7fcdf8140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcde800c500 tx=0x7fcde800c9c0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 shutdown_connections 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcdd4067b20 0x7fcdd4069fe0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcdf8153860 0x7fcdf8145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcdf80b4420 0x7fcdf81441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 --2- 172.21.15.28:0/1472511176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdf8072190 0x7fcdf8140590 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 >> 172.21.15.28:0/1472511176 conn(0x7fcdf80674f0 msgr2=0x7fcdf806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 shutdown_connections 2024-03-21T08:43:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.054+0000 7fcdff000640 1 -- 172.21.15.28:0/1472511176 wait complete. 2024-03-21T08:43:33.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.234+0000 7f7613e00640 1 Processor -- start 2024-03-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.234+0000 7f7613e00640 1 -- start start 2024-03-21T08:43:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f76140ce2f0 0x7f76140ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140cec10 con 0x7f761406c900 2024-03-21T08:43:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140d0050 con 0x7f761415ded0 2024-03-21T08:43:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76140cfe10 con 0x7f76140ce2f0 2024-03-21T08:43:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f761406c900 0x7f761406cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38622/0 (socket says 172.21.15.28:38622) 2024-03-21T08:43:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7613400640 1 -- 172.21.15.28:0/1583600919 learned_addr learned my addr 172.21.15.28:0/1583600919 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612a00640 1 --2- 172.21.15.28:0/1583600919 >> v2:172.21.15.151:3300/0 conn(0x7f76140ce2f0 0x7f76140ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161895637 0 0) 0x7f76140cec10 con 0x7f761406c900 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f8003530 con 0x7f761406c900 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880043952 0 0) 0x7f76140d0050 con 0x7f761415ded0 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76140cec10 con 0x7f761415ded0 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612a00640 1 -- 172.21.15.28:0/1583600919 >> v1:172.21.15.151:6789/0 conn(0x7f761415ded0 legacy=0x7f7614071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612a00640 1 -- 172.21.15.28:0/1583600919 >> v1:172.21.15.28:6789/0 conn(0x7f761406c900 legacy=0x7f761406cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.238+0000 7f7612a00640 1 -- 172.21.15.28:0/1583600919 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76140d0330 con 0x7f76140ce2f0 2024-03-21T08:43:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7612a00640 1 --2- 172.21.15.28:0/1583600919 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f76140ce6d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f760401c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb1c0bf550416fe server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040a66b0 con 0x7f76140ce2f0 2024-03-21T08:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76040a6850 con 0x7f76140ce2f0 2024-03-21T08:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7612000640 1 -- 172.21.15.28:0/1583600919 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040a6c00 con 0x7f76140ce2f0 2024-03-21T08:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- 172.21.15.28:0/1583600919 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 msgr2=0x7f76140ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 --2- 172.21.15.28:0/1583600919 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f76140ce6d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f760401c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- 172.21.15.28:0/1583600919 shutdown_connections 2024-03-21T08:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 --2- 172.21.15.28:0/1583600919 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f76140ce6d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- 172.21.15.28:0/1583600919 >> 172.21.15.28:0/1583600919 conn(0x7f761406fa90 msgr2=0x7f761406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- 172.21.15.28:0/1583600919 shutdown_connections 2024-03-21T08:43:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- 172.21.15.28:0/1583600919 wait complete. 2024-03-21T08:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 Processor -- start 2024-03-21T08:43:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.242+0000 7f7613e00640 1 -- start start 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f7614160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 0x7f761414af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761416bce0 con 0x7f761406c900 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761416bb60 con 0x7f76140ce2f0 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761416be60 con 0x7f761415ded0 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43340/0 (socket says 172.21.15.28:43340) 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 -- 172.21.15.28:0/2256728731 learned_addr learned my addr 172.21.15.28:0/2256728731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f760a000640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f7614160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613400640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 0x7f761414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 -- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 msgr2=0x7f761414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 0x7f761414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 -- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 msgr2=0x7f7614160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f7614160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 -- 172.21.15.28:0/2256728731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76140c5c30 con 0x7f761406c900 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613400640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 0x7f761414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7612a00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f7604019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040be070 con 0x7f761406c900 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76040a6e10 con 0x7f761406c900 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76040b8650 con 0x7f761406c900 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76140c5ed0 con 0x7f761406c900 2024-03-21T08:43:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76140c6490 con 0x7f761406c900 2024-03-21T08:43:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.246+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761404f410 con 0x7f761406c900 2024-03-21T08:43:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.250+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f76040a66d0 con 0x7f761406c900 2024-03-21T08:43:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.250+0000 7f760b400640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 0x7f75f0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.250+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f7604120dd0 con 0x7f761406c900 2024-03-21T08:43:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.250+0000 7f760a000640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 0x7f75f0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.250+0000 7f760a000640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 0x7f75f0069fe0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f761406cf70 tx=0x7f75fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:33.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.254+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7604016560 con 0x7f761406c900 2024-03-21T08:43:33.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:33.450+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f76140b0db0 con 0x7f761406c900 2024-03-21T08:43:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.070+0000 7f760b400640 1 -- 172.21.15.28:0/2256728731 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v24) v1 ==== 172+0+0 (secure 0 0 0) 0x7f76040f1720 con 0x7f761406c900 2024-03-21T08:43:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 8 and data pool 9 2024-03-21T08:43:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 msgr2=0x7f75f0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 0x7f75f0069fe0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f761406cf70 tx=0x7f75fc009040 comp rx=0 tx=0).stop 2024-03-21T08:43:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 msgr2=0x7f76141fddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f7604019680 comp rx=0 tx=0).stop 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 shutdown_connections 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f75f0067b20 0x7f75f0069fe0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f761415ded0 0x7f761414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f76140ce2f0 0x7f7614160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 --2- 172.21.15.28:0/2256728731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406c900 0x7f76141fddc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 >> 172.21.15.28:0/2256728731 conn(0x7f761406fa90 msgr2=0x7f76141523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 shutdown_connections 2024-03-21T08:43:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.086+0000 7f7613e00640 1 -- 172.21.15.28:0/2256728731 wait complete. 2024-03-21T08:43:34.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:34.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 Processor -- start 2024-03-21T08:43:34.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 -- start start 2024-03-21T08:43:34.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cc00721b0 0x7f2cc0072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 0x7f2cc0161fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc00cfdb0 con 0x7f2cc015df40 2024-03-21T08:43:34.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00adde0 con 0x7f2cc00721b0 2024-03-21T08:43:34.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc7400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0072ad0 con 0x7f2cc00ae3a0 2024-03-21T08:43:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cc00721b0 0x7f2cc0072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cbfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2cc015df40 0x7f2cc01618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38626/0 (socket says 172.21.15.28:38626) 2024-03-21T08:43:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cbfe00640 1 -- 172.21.15.28:0/3348071561 learned_addr learned my addr 172.21.15.28:0/3348071561 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.258+0000 7f2cc5400640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 0x7f2cc0161fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc4a00640 1 -- 172.21.15.28:0/3348071561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 msgr2=0x7f2cc0161fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc4a00640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 0x7f2cc0161fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc4a00640 1 -- 172.21.15.28:0/3348071561 >> v1:172.21.15.28:6789/0 conn(0x7f2cc015df40 legacy=0x7f2cc01618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc4a00640 1 -- 172.21.15.28:0/3348071561 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc006fb80 con 0x7f2cc00721b0 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc5400640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 0x7f2cc0161fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cbf400640 1 -- 172.21.15.28:0/3348071561 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803663905 0 0) 0x7f2cac001120 con 0x7f2cc015df40 2024-03-21T08:43:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc4a00640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc00721b0 0x7f2cc0072590 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40097c0 tx=0x7f2cb401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d61f0e83cbc7da server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cbf400640 1 -- 172.21.15.28:0/3348071561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40b0070 con 0x7f2cc00721b0 2024-03-21T08:43:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cbf400640 1 -- 172.21.15.28:0/3348071561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb40ab600 con 0x7f2cc00721b0 2024-03-21T08:43:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cbf400640 1 -- 172.21.15.28:0/3348071561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40ab9d0 con 0x7f2cc00721b0 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 -- 172.21.15.28:0/3348071561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc00721b0 msgr2=0x7f2cc0072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc00721b0 0x7f2cc0072590 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40097c0 tx=0x7f2cb401c410 comp rx=0 tx=0).stop 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 -- 172.21.15.28:0/3348071561 shutdown_connections 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc00ae3a0 0x7f2cc0161fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 --2- 172.21.15.28:0/3348071561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc00721b0 0x7f2cc0072590 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 -- 172.21.15.28:0/3348071561 >> 172.21.15.28:0/3348071561 conn(0x7f2cc00d0050 msgr2=0x7f2cc00d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 -- 172.21.15.28:0/3348071561 shutdown_connections 2024-03-21T08:43:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.262+0000 7f2cc7400640 1 -- 172.21.15.28:0/3348071561 wait complete. 2024-03-21T08:43:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 Processor -- start 2024-03-21T08:43:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- start start 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2cc00721b0 0x7f2cc00c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 0x7f2cc0151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0162670 con 0x7f2cc015ab90 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc01624f0 con 0x7f2cc00721b0 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00cec30 con 0x7f2cc015df40 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43366/0 (socket says 172.21.15.28:43366) 2024-03-21T08:43:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 -- 172.21.15.28:0/2823118252 learned_addr learned my addr 172.21.15.28:0/2823118252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc4a00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2cc00721b0 0x7f2cc00c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cbfe00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 0x7f2cc0151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 -- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 msgr2=0x7f2cc0151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 0x7f2cc0151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 -- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2cc00721b0 msgr2=0x7f2cc00c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2cc00721b0 0x7f2cc00c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 -- 172.21.15.28:0/2823118252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc015be80 con 0x7f2cc015ab90 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cbfe00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 0x7f2cc0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc5400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8007ef0 tx=0x7f2cb800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb801a070 con 0x7f2cc015ab90 2024-03-21T08:43:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb8015600 con 0x7f2cc015ab90 2024-03-21T08:43:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80159f0 con 0x7f2cc015ab90 2024-03-21T08:43:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc015b5e0 con 0x7f2cc015ab90 2024-03-21T08:43:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.266+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc015bba0 con 0x7f2cc015ab90 2024-03-21T08:43:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc004f410 con 0x7f2cc015ab90 2024-03-21T08:43:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2cb801d070 con 0x7f2cc015ab90 2024-03-21T08:43:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cbcc00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 0x7f2ca0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2cb8086320 con 0x7f2cc015ab90 2024-03-21T08:43:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cc4a00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 0x7f2ca0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.270+0000 7f2cc4a00640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 0x7f2ca0069fe0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40ad000 tx=0x7f2cb40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.274+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2cb8010070 con 0x7f2cc015ab90 2024-03-21T08:43:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.438+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cc015a490 con 0x7f2cc015ab90 2024-03-21T08:43:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cbcc00640 1 -- 172.21.15.28:0/2823118252 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f2cb8056540 con 0x7f2cc015ab90 2024-03-21T08:43:34.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:34.446 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:33.456671+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 msgr2=0x7f2ca0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 0x7f2ca0069fe0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40ad000 tx=0x7f2cb40a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 msgr2=0x7f2cc00acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8007ef0 tx=0x7f2cb800d4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 shutdown_connections 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cc015ab90 0x7f2cc00acec0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2ca0067b20 0x7f2ca0069fe0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2cc015df40 0x7f2cc0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 --2- 172.21.15.28:0/2823118252 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2cc00721b0 0x7f2cc00c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 >> 172.21.15.28:0/2823118252 conn(0x7f2cc00d0050 msgr2=0x7f2cc006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 shutdown_connections 2024-03-21T08:43:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.442+0000 7f2cc7400640 1 -- 172.21.15.28:0/2823118252 wait complete. 2024-03-21T08:43:34.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.602+0000 7f4852000640 1 Processor -- start 2024-03-21T08:43:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.602+0000 7f4852000640 1 -- start start 2024-03-21T08:43:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f4852000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 0x7f484c0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f4852000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f484c0b5520 0x7f484c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f4852000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c15bf00 con 0x7f484c0b34f0 2024-03-21T08:43:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f4852000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c0ce6e0 con 0x7f484c0b5520 2024-03-21T08:43:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f4852000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0cdad0 con 0x7f484c0af060 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 0x7f484c0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484aa00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f484c0b34f0 0x7f484c0b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:55968/0 (socket says 172.21.15.28:55968) 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484aa00640 1 -- 172.21.15.28:0/1462758433 learned_addr learned my addr 172.21.15.28:0/1462758433 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484be00640 1 --2- 172.21.15.28:0/1462758433 >> v2:172.21.15.151:3300/0 conn(0x7f484c0b5520 0x7f484c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484a000640 1 -- 172.21.15.28:0/1462758433 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402166515 0 0) 0x7f484c15bf00 con 0x7f484c0b34f0 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484b400640 1 -- 172.21.15.28:0/1462758433 >> v2:172.21.15.151:3300/0 conn(0x7f484c0b5520 msgr2=0x7f484c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484b400640 1 --2- 172.21.15.28:0/1462758433 >> v2:172.21.15.151:3300/0 conn(0x7f484c0b5520 0x7f484c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484b400640 1 -- 172.21.15.28:0/1462758433 >> v1:172.21.15.151:6789/0 conn(0x7f484c0b34f0 legacy=0x7f484c0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.606+0000 7f484b400640 1 -- 172.21.15.28:0/1462758433 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0cddb0 con 0x7f484c0af060 2024-03-21T08:43:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f484b400640 1 --2- 172.21.15.28:0/1462758433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 0x7f484c0cd380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f483c0097c0 tx=0x7f483c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576f65dbc75465e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f484a000640 1 -- 172.21.15.28:0/1462758433 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b0070 con 0x7f484c0af060 2024-03-21T08:43:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f484a000640 1 -- 172.21.15.28:0/1462758433 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c0ab600 con 0x7f484c0af060 2024-03-21T08:43:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f484a000640 1 -- 172.21.15.28:0/1462758433 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0ab9d0 con 0x7f484c0af060 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- 172.21.15.28:0/1462758433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 msgr2=0x7f484c0cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- 172.21.15.28:0/1462758433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 0x7f484c0cd380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f483c0097c0 tx=0x7f483c01c410 comp rx=0 tx=0).stop 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- 172.21.15.28:0/1462758433 shutdown_connections 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- 172.21.15.28:0/1462758433 >> v2:172.21.15.151:3300/0 conn(0x7f484c0b5520 0x7f484c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- 172.21.15.28:0/1462758433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c0af060 0x7f484c0cd380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- 172.21.15.28:0/1462758433 >> 172.21.15.28:0/1462758433 conn(0x7f484c0cc230 msgr2=0x7f484c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- 172.21.15.28:0/1462758433 shutdown_connections 2024-03-21T08:43:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- 172.21.15.28:0/1462758433 wait complete. 2024-03-21T08:43:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 Processor -- start 2024-03-21T08:43:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- start start 2024-03-21T08:43:34.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 0x7f484c14a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 0x7f484c149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c15d0d0 con 0x7f484c0b34f0 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c15cf50 con 0x7f484c0b5520 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.610+0000 7f4852000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c15d250 con 0x7f484c14a670 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43374/0 (socket says 172.21.15.28:43374) 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 -- 172.21.15.28:0/2754019615 learned_addr learned my addr 172.21.15.28:0/2754019615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484aa00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 0x7f484c14a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484be00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 0x7f484c149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 -- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 msgr2=0x7f484c149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 0x7f484c149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 -- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 msgr2=0x7f484c14a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 0x7f484c14a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 -- 172.21.15.28:0/2754019615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0ad450 con 0x7f484c0b34f0 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484be00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 0x7f484c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484aa00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 0x7f484c14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f484b400640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f483c0097c0 tx=0x7f483c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b0050 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c0bc070 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c0ad730 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b5830 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c0adcf0 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c14aa80 con 0x7f484c0b34f0 2024-03-21T08:43:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.614+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f483c0b59d0 con 0x7f484c0b34f0 2024-03-21T08:43:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.618+0000 7f4843400640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 0x7f482c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.618+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f483c120810 con 0x7f484c0b34f0 2024-03-21T08:43:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.618+0000 7f484aa00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 0x7f482c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.618+0000 7f484aa00640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 0x7f482c06a010 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f484c0ac950 tx=0x7f4838007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.622+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f484c14aa80 con 0x7f484c0b34f0 2024-03-21T08:43:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.782+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f484c04f410 con 0x7f484c0b34f0 2024-03-21T08:43:34.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4843400640 1 -- 172.21.15.28:0/2754019615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f483c016090 con 0x7f484c0b34f0 2024-03-21T08:43:34.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:34.788 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:33.456671+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 msgr2=0x7f482c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 0x7f482c06a010 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f484c0ac950 tx=0x7f4838007c60 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 msgr2=0x7f484c1471d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f483c0097c0 tx=0x7f483c01ac50 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 shutdown_connections 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f482c067b50 0x7f482c06a010 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f484c14a670 0x7f484c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f484c0b5520 0x7f484c14a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 --2- 172.21.15.28:0/2754019615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c0b34f0 0x7f484c1471d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 >> 172.21.15.28:0/2754019615 conn(0x7f484c0cc230 msgr2=0x7f484c154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 shutdown_connections 2024-03-21T08:43:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.786+0000 7f4852000640 1 -- 172.21.15.28:0/2754019615 wait complete. 2024-03-21T08:43:34.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-21T08:43:34.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.950+0000 7f2a3d800640 1 Processor -- start 2024-03-21T08:43:34.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.950+0000 7f2a3d800640 1 -- start start 2024-03-21T08:43:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3d800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a380cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3d800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f2a380ce320 0x7f2a380ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a380b11e0 con 0x7f2a3815ba50 2024-03-21T08:43:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3d800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a380cc6b0 con 0x7f2a380ce320 2024-03-21T08:43:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3d800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380ccea0 con 0x7f2a380b0da0 2024-03-21T08:43:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2a3815ba50 0x7f2a38071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38640/0 (socket says 172.21.15.28:38640) 2024-03-21T08:43:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37e00640 1 -- 172.21.15.28:0/1689853196 learned_addr learned my addr 172.21.15.28:0/1689853196 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a380cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3ce00640 1 --2- 172.21.15.28:0/1689853196 >> v2:172.21.15.151:3300/0 conn(0x7f2a380ce320 0x7f2a380ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492327534 0 0) 0x7f2a380b11e0 con 0x7f2a3815ba50 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c003530 con 0x7f2a3815ba50 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1476142016 0 0) 0x7f2a1c003530 con 0x7f2a3815ba50 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 msgr2=0x7f2a380cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a380cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 >> v2:172.21.15.151:3300/0 conn(0x7f2a380ce320 msgr2=0x7f2a380ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 --2- 172.21.15.28:0/1689853196 >> v2:172.21.15.151:3300/0 conn(0x7f2a380ce320 0x7f2a380ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3815be30 con 0x7f2a3815ba50 2024-03-21T08:43:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f2a28001680 con 0x7f2a3815ba50 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 >> v1:172.21.15.28:6789/0 conn(0x7f2a3815ba50 legacy=0x7f2a38071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a1c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a1c004d50 0x7f2a1c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380ccea0 con 0x7f2a1c003f90 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c00b9d0 con 0x7f2a1c004d50 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c00bb50 con 0x7f2a380b0da0 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a37e00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a1c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3ce00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a1c004d50 0x7f2a1c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 -- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 msgr2=0x7f2a1c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380b0da0 0x7f2a1c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 -- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a1c004d50 msgr2=0x7f2a1c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a1c004d50 0x7f2a1c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a2fe00640 1 -- 172.21.15.28:0/1689853196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c003530 con 0x7f2a1c003f90 2024-03-21T08:43:34.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.954+0000 7f2a3ce00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a1c004d50 0x7f2a1c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a1c004810 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009b10 tx=0x7f2a2001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b0d8b70606fdf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a200b2070 con 0x7f2a1c003f90 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a200a5660 con 0x7f2a1c003f90 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a37400640 1 -- 172.21.15.28:0/1689853196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a200096b0 con 0x7f2a1c003f90 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 -- 172.21.15.28:0/1689853196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 msgr2=0x7f2a1c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 --2- 172.21.15.28:0/1689853196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a1c004810 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009b10 tx=0x7f2a2001ca70 comp rx=0 tx=0).stop 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3ce00640 1 -- 172.21.15.28:0/1689853196 reap_dead start 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 -- 172.21.15.28:0/1689853196 shutdown_connections 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 -- 172.21.15.28:0/1689853196 >> 172.21.15.28:0/1689853196 conn(0x7f2a3806fa90 msgr2=0x7f2a3806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:34.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 -- 172.21.15.28:0/1689853196 shutdown_connections 2024-03-21T08:43:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 -- 172.21.15.28:0/1689853196 wait complete. 2024-03-21T08:43:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.958+0000 7f2a3d800640 1 Processor -- start 2024-03-21T08:43:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- start start 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 0x7f2a380acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 0x7f2a3815e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380713b0 con 0x7f2a1c003f90 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a380ccbc0 con 0x7f2a3815a090 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a38071690 con 0x7f2a380ad010 2024-03-21T08:43:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43390/0 (socket says 172.21.15.28:43390) 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 -- 172.21.15.28:0/101816770 learned_addr learned my addr 172.21.15.28:0/101816770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 0x7f2a380acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3ce00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 0x7f2a3815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 -- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 msgr2=0x7f2a3815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 0x7f2a3815e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 -- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 msgr2=0x7f2a380acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 0x7f2a380acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 -- 172.21.15.28:0/101816770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a380afb90 con 0x7f2a1c003f90 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3ce00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 0x7f2a3815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 0x7f2a380acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a37e00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2a28012470 tx=0x7f2a28012930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2801c070 con 0x7f2a1c003f90 2024-03-21T08:43:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a28014830 con 0x7f2a1c003f90 2024-03-21T08:43:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a380afe70 con 0x7f2a1c003f90 2024-03-21T08:43:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28014c20 con 0x7f2a1c003f90 2024-03-21T08:43:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a380b0430 con 0x7f2a1c003f90 2024-03-21T08:43:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.962+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3804f410 con 0x7f2a1c003f90 2024-03-21T08:43:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.966+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f2a2801f070 con 0x7f2a1c003f90 2024-03-21T08:43:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.966+0000 7f2a34c00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 0x7f2a1406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.966+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2a28088c20 con 0x7f2a1c003f90 2024-03-21T08:43:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.966+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 0x7f2a1406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.966+0000 7f2a2fe00640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 0x7f2a1406a010 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2a200af000 tx=0x7f2a200046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:34.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:34.970+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2a28059490 con 0x7f2a1c003f90 2024-03-21T08:43:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:35.174+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2a380701a0 con 0x7f2a1c003f90 2024-03-21T08:43:36.046 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-03-21T08:43:36.042+0000 7ff168e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:43:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.078+0000 7f2a34c00640 1 -- 172.21.15.28:0/101816770 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2a28058d90 con 0x7f2a1c003f90 2024-03-21T08:43:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 msgr2=0x7f2a1406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 0x7f2a1406a010 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2a200af000 tx=0x7f2a200046d0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 msgr2=0x7f2a380ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2a28012470 tx=0x7f2a28012930 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 shutdown_connections 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f2a14067b50 0x7f2a1406a010 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2a380ad010 0x7f2a3815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f2a3815a090 0x7f2a380acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 --2- 172.21.15.28:0/101816770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a1c003f90 0x7f2a380ca5d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 >> 172.21.15.28:0/101816770 conn(0x7f2a3806fa90 msgr2=0x7f2a380b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 shutdown_connections 2024-03-21T08:43:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.082+0000 7f2a3d800640 1 -- 172.21.15.28:0/101816770 wait complete. 2024-03-21T08:43:36.103 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-21T08:43:36.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-21T08:43:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.242+0000 7fce56a00640 1 Processor -- start 2024-03-21T08:43:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.242+0000 7fce56a00640 1 -- start start 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce5015ded0 0x7fce5006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 0x7fce5015f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 0x7fce50161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5015eb70 con 0x7fce5015ded0 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5015e2b0 con 0x7fce500cfef0 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500d0690 con 0x7fce50161880 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce5015ded0 0x7fce5006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce5015ded0 0x7fce5006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43392/0 (socket says 172.21.15.28:43392) 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 -- 172.21.15.28:0/2539885049 learned_addr learned my addr 172.21.15.28:0/2539885049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56000640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 0x7fce50161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce54c00640 1 --2- 172.21.15.28:0/2539885049 >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 0x7fce5015f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 -- 172.21.15.28:0/2539885049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 msgr2=0x7fce50161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 0x7fce50161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 -- 172.21.15.28:0/2539885049 >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 msgr2=0x7fce5015f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 --2- 172.21.15.28:0/2539885049 >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 0x7fce5015f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce55600640 1 -- 172.21.15.28:0/2539885049 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500d03b0 con 0x7fce5015ded0 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce56000640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 0x7fce50161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.246+0000 7fce54c00640 1 --2- 172.21.15.28:0/2539885049 >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 0x7fce5015f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce55600640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce5015ded0 0x7fce5006c280 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fce500cf490 tx=0x7fce4401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c484c68ec2ae28 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce4fe00640 1 -- 172.21.15.28:0/2539885049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440b0070 con 0x7fce5015ded0 2024-03-21T08:43:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce4fe00640 1 -- 172.21.15.28:0/2539885049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce440ab600 con 0x7fce5015ded0 2024-03-21T08:43:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce4fe00640 1 -- 172.21.15.28:0/2539885049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440ab9d0 con 0x7fce5015ded0 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- 172.21.15.28:0/2539885049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce5015ded0 msgr2=0x7fce5006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce5015ded0 0x7fce5006c280 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fce500cf490 tx=0x7fce4401c490 comp rx=0 tx=0).stop 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- 172.21.15.28:0/2539885049 shutdown_connections 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce50161880 0x7fce50161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- 172.21.15.28:0/2539885049 >> v2:172.21.15.151:3300/0 conn(0x7fce500cfef0 0x7fce5015f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- 172.21.15.28:0/2539885049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce5015ded0 0x7fce5006c280 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- 172.21.15.28:0/2539885049 >> 172.21.15.28:0/2539885049 conn(0x7fce5006fa90 msgr2=0x7fce5006fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:36.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- 172.21.15.28:0/2539885049 shutdown_connections 2024-03-21T08:43:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- 172.21.15.28:0/2539885049 wait complete. 2024-03-21T08:43:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 Processor -- start 2024-03-21T08:43:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- start start 2024-03-21T08:43:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce500cfef0 0x7fce500ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce5015ded0 0x7fce5014ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5006d300 con 0x7fce50161880 2024-03-21T08:43:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5006d180 con 0x7fce5015ded0 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.250+0000 7fce56a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5006d480 con 0x7fce500cfef0 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43406/0 (socket says 172.21.15.28:43406) 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 -- 172.21.15.28:0/1022919932 learned_addr learned my addr 172.21.15.28:0/1022919932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce54c00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce5015ded0 0x7fce5014ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce55600640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce500cfef0 0x7fce500ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 -- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce500cfef0 msgr2=0x7fce500ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce500cfef0 0x7fce500ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 -- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce5015ded0 msgr2=0x7fce5014ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce5015ded0 0x7fce5014ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 -- 172.21.15.28:0/1022919932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500c93c0 con 0x7fce50161880 2024-03-21T08:43:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56000640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fce4800cf20 tx=0x7fce4800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4801a070 con 0x7fce50161880 2024-03-21T08:43:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4800d840 con 0x7fce50161880 2024-03-21T08:43:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce500c96a0 con 0x7fce50161880 2024-03-21T08:43:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4800dc30 con 0x7fce50161880 2024-03-21T08:43:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce500c9c60 con 0x7fce50161880 2024-03-21T08:43:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5004f410 con 0x7fce50161880 2024-03-21T08:43:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.254+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fce4801d070 con 0x7fce50161880 2024-03-21T08:43:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.258+0000 7fce4d600640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 0x7fce3006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.258+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce48086870 con 0x7fce50161880 2024-03-21T08:43:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.258+0000 7fce55600640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 0x7fce3006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.258+0000 7fce55600640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 0x7fce3006a010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce500d2940 tx=0x7fce44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.262+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fce48056a90 con 0x7fce50161880 2024-03-21T08:43:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:36.466+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fce500b7890 con 0x7fce50161880 2024-03-21T08:43:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce4d600640 1 -- 172.21.15.28:0/1022919932 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fce4805ab30 con 0x7fce50161880 2024-03-21T08:43:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 msgr2=0x7fce3006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 0x7fce3006a010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce500d2940 tx=0x7fce44008040 comp rx=0 tx=0).stop 2024-03-21T08:43:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 msgr2=0x7fce500c89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fce4800cf20 tx=0x7fce4800a7d0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 shutdown_connections 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce50161880 0x7fce500c89e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fce5015ded0 0x7fce5014ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fce30067b50 0x7fce3006a010 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 --2- 172.21.15.28:0/1022919932 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fce500cfef0 0x7fce500ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 >> 172.21.15.28:0/1022919932 conn(0x7fce5006fa90 msgr2=0x7fce50071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 shutdown_connections 2024-03-21T08:43:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.082+0000 7fce56a00640 1 -- 172.21.15.28:0/1022919932 wait complete. 2024-03-21T08:43:37.105 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-21T08:43:37.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.242+0000 7fec44c00640 1 Processor -- start 2024-03-21T08:43:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.242+0000 7fec44c00640 1 -- start start 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec44c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec4009a010 0x7fec400d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec44c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400cc1d0 con 0x7fec4015e0c0 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec44c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4009a480 con 0x7fec40099b80 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec44c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec400d1730 con 0x7fec4009a010 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec37e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fec40099b80 0x7fec400cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:55984/0 (socket says 172.21.15.28:55984) 2024-03-21T08:43:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fec4015e0c0 0x7fec4006cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38656/0 (socket says 172.21.15.28:38656) 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec37e00640 1 -- 172.21.15.28:0/237459646 learned_addr learned my addr 172.21.15.28:0/237459646 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3e800640 1 --2- 172.21.15.28:0/237459646 >> v2:172.21.15.28:3300/0 conn(0x7fec4009a010 0x7fec400d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3e800640 1 -- 172.21.15.28:0/237459646 >> v1:172.21.15.151:6789/0 conn(0x7fec40099b80 legacy=0x7fec400cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3e800640 1 -- 172.21.15.28:0/237459646 >> v1:172.21.15.28:6789/0 conn(0x7fec4015e0c0 legacy=0x7fec4006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3e800640 1 -- 172.21.15.28:0/237459646 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400d1450 con 0x7fec4009a010 2024-03-21T08:43:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.246+0000 7fec3d400640 1 -- 172.21.15.28:0/237459646 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007173741 0 0) 0x7fec4009a480 con 0x7fec40099b80 2024-03-21T08:43:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec3e800640 1 --2- 172.21.15.28:0/237459646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4009a010 0x7fec400d03c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fec38009d90 tx=0x7fec3801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3057e87b75611b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec3d400640 1 -- 172.21.15.28:0/237459646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec380b2070 con 0x7fec4009a010 2024-03-21T08:43:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec3d400640 1 -- 172.21.15.28:0/237459646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec380a5a00 con 0x7fec4009a010 2024-03-21T08:43:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec3d400640 1 -- 172.21.15.28:0/237459646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec380a5dd0 con 0x7fec4009a010 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- 172.21.15.28:0/237459646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4009a010 msgr2=0x7fec400d03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 --2- 172.21.15.28:0/237459646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4009a010 0x7fec400d03c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fec38009d90 tx=0x7fec3801c940 comp rx=0 tx=0).stop 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- 172.21.15.28:0/237459646 shutdown_connections 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 --2- 172.21.15.28:0/237459646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4009a010 0x7fec400d03c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- 172.21.15.28:0/237459646 >> 172.21.15.28:0/237459646 conn(0x7fec400b34d0 msgr2=0x7fec400b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- 172.21.15.28:0/237459646 shutdown_connections 2024-03-21T08:43:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- 172.21.15.28:0/237459646 wait complete. 2024-03-21T08:43:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 Processor -- start 2024-03-21T08:43:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.250+0000 7fec44c00640 1 -- start start 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec4009a010 0x7fec400c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec4015e0c0 0x7fec400c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400cde00 con 0x7fec40099b80 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400cdc80 con 0x7fec4015e0c0 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4016bca0 con 0x7fec4009a010 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43424/0 (socket says 172.21.15.28:43424) 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 -- 172.21.15.28:0/1439496035 learned_addr learned my addr 172.21.15.28:0/1439496035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec37e00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec4009a010 0x7fec400c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3e800640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec4015e0c0 0x7fec400c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 -- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec4009a010 msgr2=0x7fec400c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec4009a010 0x7fec400c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 -- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec4015e0c0 msgr2=0x7fec400c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec4015e0c0 0x7fec400c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 -- 172.21.15.28:0/1439496035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec401539b0 con 0x7fec40099b80 2024-03-21T08:43:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec3de00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c420 tx=0x7fec3000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300074e0 con 0x7fec40099b80 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30007680 con 0x7fec40099b80 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec40153c90 con 0x7fec40099b80 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30004a30 con 0x7fec40099b80 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec40154250 con 0x7fec40099b80 2024-03-21T08:43:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.254+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4004f410 con 0x7fec40099b80 2024-03-21T08:43:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.258+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fec3001a020 con 0x7fec40099b80 2024-03-21T08:43:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.258+0000 7fec36000640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 0x7fec1c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.258+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec30085900 con 0x7fec40099b80 2024-03-21T08:43:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.258+0000 7fec37e00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 0x7fec1c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.258+0000 7fec37e00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 0x7fec1c069fe0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec40154650 tx=0x7fec28007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.262+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fec30056180 con 0x7fec40099b80 2024-03-21T08:43:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.458+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec400b2b10 con 0x7fec40099b80 2024-03-21T08:43:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.458+0000 7fec36000640 1 -- 172.21.15.28:0/1439496035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fec30055a80 con 0x7fec40099b80 2024-03-21T08:43:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:37.461 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:43:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:43:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 msgr2=0x7fec1c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 0x7fec1c069fe0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec40154650 tx=0x7fec28007c10 comp rx=0 tx=0).stop 2024-03-21T08:43:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 msgr2=0x7fec401517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c420 tx=0x7fec3000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 shutdown_connections 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fec4015e0c0 0x7fec400c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fec1c067b20 0x7fec1c069fe0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fec4009a010 0x7fec400c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 --2- 172.21.15.28:0/1439496035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40099b80 0x7fec401517f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 >> 172.21.15.28:0/1439496035 conn(0x7fec400b34d0 msgr2=0x7fec4006cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 shutdown_connections 2024-03-21T08:43:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.462+0000 7fec44c00640 1 -- 172.21.15.28:0/1439496035 wait complete. 2024-03-21T08:43:37.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 Processor -- start 2024-03-21T08:43:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 -- start start 2024-03-21T08:43:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f840aff70 0x7f1f8406b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840d0540 con 0x7f1f8406eb30 2024-03-21T08:43:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840cd6e0 con 0x7f1f8415dd00 2024-03-21T08:43:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8c400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840cd9c0 con 0x7f1f840aff70 2024-03-21T08:43:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8a400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f8406eb30 0x7f1f84161a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38662/0 (socket says 172.21.15.28:38662) 2024-03-21T08:43:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f89000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f840aff70 0x7f1f8406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.622+0000 7f1f8a400640 1 -- 172.21.15.28:0/1526775340 learned_addr learned my addr 172.21.15.28:0/1526775340 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f89000640 1 -- 172.21.15.28:0/1526775340 >> v1:172.21.15.151:6789/0 conn(0x7f1f8415dd00 legacy=0x7f1f840cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f89000640 1 -- 172.21.15.28:0/1526775340 >> v1:172.21.15.28:6789/0 conn(0x7f1f8406eb30 legacy=0x7f1f84161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f89000640 1 -- 172.21.15.28:0/1526775340 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840cdca0 con 0x7f1f840aff70 2024-03-21T08:43:37.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f89000640 1 --2- 172.21.15.28:0/1526775340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f840aff70 0x7f1f8406b170 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1f700097c0 tx=0x7f1f7001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8defb0f8af04bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f83e00640 1 -- 172.21.15.28:0/1526775340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700b0070 con 0x7f1f840aff70 2024-03-21T08:43:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f83e00640 1 -- 172.21.15.28:0/1526775340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f700ab600 con 0x7f1f840aff70 2024-03-21T08:43:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f83e00640 1 -- 172.21.15.28:0/1526775340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700ab9d0 con 0x7f1f840aff70 2024-03-21T08:43:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f8c400640 1 -- 172.21.15.28:0/1526775340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f840aff70 msgr2=0x7f1f8406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.626+0000 7f1f8c400640 1 --2- 172.21.15.28:0/1526775340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f840aff70 0x7f1f8406b170 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1f700097c0 tx=0x7f1f7001c430 comp rx=0 tx=0).stop 2024-03-21T08:43:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- 172.21.15.28:0/1526775340 shutdown_connections 2024-03-21T08:43:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 --2- 172.21.15.28:0/1526775340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f840aff70 0x7f1f8406b170 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- 172.21.15.28:0/1526775340 >> 172.21.15.28:0/1526775340 conn(0x7f1f84099a30 msgr2=0x7f1f84099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- 172.21.15.28:0/1526775340 shutdown_connections 2024-03-21T08:43:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- 172.21.15.28:0/1526775340 wait complete. 2024-03-21T08:43:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 Processor -- start 2024-03-21T08:43:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- start start 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f8406eb30 0x7f1f841582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f840aff70 0x7f1f84155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8416be70 con 0x7f1f8415dd00 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8416bcf0 con 0x7f1f840aff70 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8416bff0 con 0x7f1f8406eb30 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f89000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f840aff70 0x7f1f84155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43434/0 (socket says 172.21.15.28:43434) 2024-03-21T08:43:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 -- 172.21.15.28:0/2408739834 learned_addr learned my addr 172.21.15.28:0/2408739834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f89a00640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f8406eb30 0x7f1f841582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 -- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f8406eb30 msgr2=0x7f1f841582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f8406eb30 0x7f1f841582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 -- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f840aff70 msgr2=0x7f1f84155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f840aff70 0x7f1f84155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 -- 172.21.15.28:0/2408739834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84158810 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.630+0000 7f1f8a400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00cbf0 tx=0x7f1f7c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c017070 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c00ad40 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c012600 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f84157230 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f841577f0 con 0x7f1f8415dd00 2024-03-21T08:43:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8404f410 con 0x7f1f8415dd00 2024-03-21T08:43:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1f7c01a070 con 0x7f1f8415dd00 2024-03-21T08:43:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.634+0000 7f1f81600640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 0x7f1f6406a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.638+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1f7c085e30 con 0x7f1f8415dd00 2024-03-21T08:43:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.638+0000 7f1f89a00640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 0x7f1f6406a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.638+0000 7f1f89a00640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 0x7f1f6406a030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f780098e0 tx=0x7f1f78009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.638+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1f7c055fb0 con 0x7f1f8415dd00 2024-03-21T08:43:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.806+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f840cd4c0 con 0x7f1f8415dd00 2024-03-21T08:43:37.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.806+0000 7f1f81600640 1 -- 172.21.15.28:0/2408739834 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f1f7c05a050 con 0x7f1f8415dd00 2024-03-21T08:43:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:37.810 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 msgr2=0x7f1f6406a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 0x7f1f6406a030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f780098e0 tx=0x7f1f78009060 comp rx=0 tx=0).stop 2024-03-21T08:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 msgr2=0x7f1f84160c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00cbf0 tx=0x7f1f7c00a430 comp rx=0 tx=0).stop 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 shutdown_connections 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f8415dd00 0x7f1f84160c20 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1f840aff70 0x7f1f84155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1f64067b70 0x7f1f6406a030 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 --2- 172.21.15.28:0/2408739834 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1f8406eb30 0x7f1f841582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 >> 172.21.15.28:0/2408739834 conn(0x7f1f84099a30 msgr2=0x7f1f8406ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 shutdown_connections 2024-03-21T08:43:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.810+0000 7f1f8c400640 1 -- 172.21.15.28:0/2408739834 wait complete. 2024-03-21T08:43:37.831 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-21T08:43:37.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:43:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 Processor -- start 2024-03-21T08:43:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 -- start start 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff7fc099b80 0x7ff7fc09a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0cc1d0 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0d1450 con 0x7ff7fc0cfcb0 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff803200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7fc0d1730 con 0x7ff7fc099b80 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff7fbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff7fc15e0c0 0x7ff7fc06cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38672/0 (socket says 172.21.15.28:38672) 2024-03-21T08:43:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.970+0000 7ff7fbe00640 1 -- 172.21.15.28:0/940529337 learned_addr learned my addr 172.21.15.28:0/940529337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7f3400640 1 --2- 172.21.15.28:0/940529337 >> v2:172.21.15.151:3300/0 conn(0x7ff7fc099b80 0x7ff7fc09a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825263109 0 0) 0x7ff7fc0cc1d0 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003530 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226203714 0 0) 0x7ff7fc0d1450 con 0x7ff7fc0cfcb0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc0cc1d0 con 0x7ff7fc0cfcb0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780769842 0 0) 0x7ff7e0003530 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 >> v2:172.21.15.151:3300/0 conn(0x7ff7fc099b80 msgr2=0x7ff7fc09a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 --2- 172.21.15.28:0/940529337 >> v2:172.21.15.151:3300/0 conn(0x7ff7fc099b80 0x7ff7fc09a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 >> v1:172.21.15.151:6789/0 conn(0x7ff7fc0cfcb0 legacy=0x7ff7fc0d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0cc470 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7ff7ec001680 con 0x7ff7fc15e0c0 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 >> v1:172.21.15.28:6789/0 conn(0x7ff7fc15e0c0 legacy=0x7ff7fc06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e0003ba0 0x7ff7e0003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e0004810 0x7ff7e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 0x7ff7e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc0d1730 con 0x7ff7e00051d0 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e000c0a0 con 0x7ff7e0003ba0 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e000c2a0 con 0x7ff7e0004810 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 0x7ff7e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7fbe00640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e0003ba0 0x7ff7e0003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff7f3400640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e0004810 0x7ff7e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 -- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e0004810 msgr2=0x7ff7e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7e0004810 0x7ff7e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 -- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e0003ba0 msgr2=0x7ff7e0003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e0003ba0 0x7ff7e0003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.974+0000 7ff801200640 1 -- 172.21.15.28:0/940529337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0003530 con 0x7ff7e00051d0 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff801200640 1 -- 172.21.15.28:0/940529337 reap_dead start 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff801200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 0x7ff7e000b9d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4001fd0 tx=0x7ff7f401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e300e78918754f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4001c20 con 0x7ff7e00051d0 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f4003980 con 0x7ff7e00051d0 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff7fb400640 1 -- 172.21.15.28:0/940529337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4003d70 con 0x7ff7e00051d0 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 msgr2=0x7ff7e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 0x7ff7e000b9d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4001fd0 tx=0x7ff7f401c900 comp rx=0 tx=0).stop 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- 172.21.15.28:0/940529337 shutdown_connections 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 --2- 172.21.15.28:0/940529337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7e00051d0 0x7ff7e000b9d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- 172.21.15.28:0/940529337 >> 172.21.15.28:0/940529337 conn(0x7ff7fc0b34d0 msgr2=0x7ff7fc0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- 172.21.15.28:0/940529337 shutdown_connections 2024-03-21T08:43:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- 172.21.15.28:0/940529337 wait complete. 2024-03-21T08:43:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 Processor -- start 2024-03-21T08:43:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- start start 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e00051d0 0x7ff7fc154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7fc0c8ab0 0x7ff7fc0c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc09a010 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc15e4a0 con 0x7ff7e00051d0 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff803200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc09a190 con 0x7ff7fc0c8ab0 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.978+0000 7ff7f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43452/0 (socket says 172.21.15.28:43452) 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 -- 172.21.15.28:0/1101776455 learned_addr learned my addr 172.21.15.28:0/1101776455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7fbe00640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e00051d0 0x7ff7fc154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff801200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7fc0c8ab0 0x7ff7fc0c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 -- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7fc0c8ab0 msgr2=0x7ff7fc0c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7fc0c8ab0 0x7ff7fc0c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 -- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e00051d0 msgr2=0x7ff7fc154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e00051d0 0x7ff7fc154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 -- 172.21.15.28:0/1101776455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc155370 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f3400640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4011960 tx=0x7ff7e4011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e400faf0 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e400fc90 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4014440 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc0b6060 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc0b62e0 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc04f410 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff7e4012040 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.982+0000 7ff7f8c00640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 0x7ff7d8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.986+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7e4086670 con 0x7ff7fc0b5be0 2024-03-21T08:43:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.986+0000 7ff7fbe00640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 0x7ff7d8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.986+0000 7ff7fbe00640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 0x7ff7d8069fe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec001000 tx=0x7ff7ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:37.986+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff7fc0b6060 con 0x7ff7fc0b5be0 2024-03-21T08:43:38.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:43:38.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:43:38.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.150+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7fc15d280 con 0x7ff7d8067b20 2024-03-21T08:43:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.158+0000 7ff7f8c00640 1 -- 172.21.15.28:0/1101776455 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+248843 (secure 0 0 0) 0x7ff7fc15d280 con 0x7ff7d8067b20 2024-03-21T08:43:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:43:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 msgr2=0x7ff7d8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 0x7ff7d8069fe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec001000 tx=0x7ff7ec009040 comp rx=0 tx=0).stop 2024-03-21T08:43:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 msgr2=0x7ff7fc1542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4011960 tx=0x7ff7e4011e20 comp rx=0 tx=0).stop 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 shutdown_connections 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff7d8067b20 0x7ff7d8069fe0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff7fc0c8ab0 0x7ff7fc0c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7fc0b5be0 0x7ff7fc1542b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 --2- 172.21.15.28:0/1101776455 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff7e00051d0 0x7ff7fc154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 >> 172.21.15.28:0/1101776455 conn(0x7ff7fc0b34d0 msgr2=0x7ff7fc0d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 shutdown_connections 2024-03-21T08:43:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.162+0000 7ff803200640 1 -- 172.21.15.28:0/1101776455 wait complete. 2024-03-21T08:43:38.184 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-21T08:43:37.753502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":178,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238812,"kb_used_data":5996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735908,"statfs":{"total":773094113280,"available":772849569792,"internally_reserved":0,"allocated":6139904,"data_stored":3388164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705743"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.105337+0000","last_change":"2024-03-21T08:43:34.104498+0000","last_active":"2024-03-21T08:43:34.105337+0000","last_peered":"2024-03-21T08:43:34.105337+0000","last_clean":"2024-03-21T08:43:34.105337+0000","last_became_active":"2024-03-21T08:43:34.102336+0000","last_became_peered":"2024-03-21T08:43:34.102336+0000","last_unstale":"2024-03-21T08:43:34.105337+0000","last_undegraded":"2024-03-21T08:43:34.105337+0000","last_fullsized":"2024-03-21T08:43:34.105337+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:44:57.769559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.066283+0000","last_change":"2024-03-21T08:43:33.066283+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.066283+0000","last_undegraded":"2024-03-21T08:43:33.066283+0000","last_fullsized":"2024-03-21T08:43:33.066283+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084512+0000","last_change":"2024-03-21T08:43:34.079798+0000","last_active":"2024-03-21T08:43:34.084512+0000","last_peered":"2024-03-21T08:43:34.084512+0000","last_clean":"2024-03-21T08:43:34.084512+0000","last_became_active":"2024-03-21T08:43:34.079557+0000","last_became_peered":"2024-03-21T08:43:34.079557+0000","last_unstale":"2024-03-21T08:43:34.084512+0000","last_undegraded":"2024-03-21T08:43:34.084512+0000","last_fullsized":"2024-03-21T08:43:34.084512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:33:45.824524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065481+0000","last_change":"2024-03-21T08:43:33.065481+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065481+0000","last_undegraded":"2024-03-21T08:43:33.065481+0000","last_fullsized":"2024-03-21T08:43:33.065481+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074891+0000","last_change":"2024-03-21T08:43:34.069862+0000","last_active":"2024-03-21T08:43:34.074891+0000","last_peered":"2024-03-21T08:43:34.074891+0000","last_clean":"2024-03-21T08:43:34.074891+0000","last_became_active":"2024-03-21T08:43:34.069561+0000","last_became_peered":"2024-03-21T08:43:34.069561+0000","last_unstale":"2024-03-21T08:43:34.074891+0000","last_undegraded":"2024-03-21T08:43:34.074891+0000","last_fullsized":"2024-03-21T08:43:34.074891+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:02:24.282774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064693+0000","last_change":"2024-03-21T08:43:33.064693+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064693+0000","last_undegraded":"2024-03-21T08:43:33.064693+0000","last_fullsized":"2024-03-21T08:43:33.064693+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065768+0000","last_change":"2024-03-21T08:43:33.065768+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065768+0000","last_undegraded":"2024-03-21T08:43:33.065768+0000","last_fullsized":"2024-03-21T08:43:33.065768+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062573+0000","last_change":"2024-03-21T08:43:33.062573+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062573+0000","last_undegraded":"2024-03-21T08:43:33.062573+0000","last_fullsized":"2024-03-21T08:43:33.062573+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063862+0000","last_change":"2024-03-21T08:43:33.063862+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063862+0000","last_undegraded":"2024-03-21T08:43:33.063862+0000","last_fullsized":"2024-03-21T08:43:33.063862+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079493+0000","last_change":"2024-03-21T08:43:34.076812+0000","last_active":"2024-03-21T08:43:34.079493+0000","last_peered":"2024-03-21T08:43:34.079493+0000","last_clean":"2024-03-21T08:43:34.079493+0000","last_became_active":"2024-03-21T08:43:34.076556+0000","last_became_peered":"2024-03-21T08:43:34.076556+0000","last_unstale":"2024-03-21T08:43:34.079493+0000","last_undegraded":"2024-03-21T08:43:34.079493+0000","last_fullsized":"2024-03-21T08:43:34.079493+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:17:35.654015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.099722+0000","last_change":"2024-03-21T08:43:34.096375+0000","last_active":"2024-03-21T08:43:34.099722+0000","last_peered":"2024-03-21T08:43:34.099722+0000","last_clean":"2024-03-21T08:43:34.099722+0000","last_became_active":"2024-03-21T08:43:34.095943+0000","last_became_peered":"2024-03-21T08:43:34.095943+0000","last_unstale":"2024-03-21T08:43:34.099722+0000","last_undegraded":"2024-03-21T08:43:34.099722+0000","last_fullsized":"2024-03-21T08:43:34.099722+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:20.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065944+0000","last_change":"2024-03-21T08:43:33.065944+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065944+0000","last_undegraded":"2024-03-21T08:43:33.065944+0000","last_fullsized":"2024-03-21T08:43:33.065944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094467+0000","last_change":"2024-03-21T08:43:34.090738+0000","last_active":"2024-03-21T08:43:34.094467+0000","last_peered":"2024-03-21T08:43:34.094467+0000","last_clean":"2024-03-21T08:43:34.094467+0000","last_became_active":"2024-03-21T08:43:34.090493+0000","last_became_peered":"2024-03-21T08:43:34.090493+0000","last_unstale":"2024-03-21T08:43:34.094467+0000","last_undegraded":"2024-03-21T08:43:34.094467+0000","last_fullsized":"2024-03-21T08:43:34.094467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063655+0000","last_change":"2024-03-21T08:43:33.063655+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063655+0000","last_undegraded":"2024-03-21T08:43:33.063655+0000","last_fullsized":"2024-03-21T08:43:33.063655+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063371+0000","last_change":"2024-03-21T08:43:33.063371+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063371+0000","last_undegraded":"2024-03-21T08:43:33.063371+0000","last_fullsized":"2024-03-21T08:43:33.063371+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065030+0000","last_change":"2024-03-21T08:43:33.065030+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065030+0000","last_undegraded":"2024-03-21T08:43:33.065030+0000","last_fullsized":"2024-03-21T08:43:33.065030+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068546+0000","last_change":"2024-03-21T08:43:33.068546+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068546+0000","last_undegraded":"2024-03-21T08:43:33.068546+0000","last_fullsized":"2024-03-21T08:43:33.068546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093933+0000","last_change":"2024-03-21T08:43:34.090757+0000","last_active":"2024-03-21T08:43:34.093933+0000","last_peered":"2024-03-21T08:43:34.093933+0000","last_clean":"2024-03-21T08:43:34.093933+0000","last_became_active":"2024-03-21T08:43:34.090125+0000","last_became_peered":"2024-03-21T08:43:34.090125+0000","last_unstale":"2024-03-21T08:43:34.093933+0000","last_undegraded":"2024-03-21T08:43:34.093933+0000","last_fullsized":"2024-03-21T08:43:34.093933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:14:11.321572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.103454+0000","last_change":"2024-03-21T08:43:34.099533+0000","last_active":"2024-03-21T08:43:34.103454+0000","last_peered":"2024-03-21T08:43:34.103454+0000","last_clean":"2024-03-21T08:43:34.103454+0000","last_became_active":"2024-03-21T08:43:34.099273+0000","last_became_peered":"2024-03-21T08:43:34.099273+0000","last_unstale":"2024-03-21T08:43:34.103454+0000","last_undegraded":"2024-03-21T08:43:34.103454+0000","last_fullsized":"2024-03-21T08:43:34.103454+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:25:59.522961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.070108+0000","last_change":"2024-03-21T08:43:33.070108+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.070108+0000","last_undegraded":"2024-03-21T08:43:33.070108+0000","last_fullsized":"2024-03-21T08:43:33.070108+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073021+0000","last_change":"2024-03-21T08:43:34.069515+0000","last_active":"2024-03-21T08:43:34.073021+0000","last_peered":"2024-03-21T08:43:34.073021+0000","last_clean":"2024-03-21T08:43:34.073021+0000","last_became_active":"2024-03-21T08:43:34.069239+0000","last_became_peered":"2024-03-21T08:43:34.069239+0000","last_unstale":"2024-03-21T08:43:34.073021+0000","last_undegraded":"2024-03-21T08:43:34.073021+0000","last_fullsized":"2024-03-21T08:43:34.073021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:40:59.946986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064406+0000","last_change":"2024-03-21T08:43:33.064406+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064406+0000","last_undegraded":"2024-03-21T08:43:33.064406+0000","last_fullsized":"2024-03-21T08:43:33.064406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100244+0000","last_change":"2024-03-21T08:43:34.097093+0000","last_active":"2024-03-21T08:43:34.100244+0000","last_peered":"2024-03-21T08:43:34.100244+0000","last_clean":"2024-03-21T08:43:34.100244+0000","last_became_active":"2024-03-21T08:43:34.096448+0000","last_became_peered":"2024-03-21T08:43:34.096448+0000","last_unstale":"2024-03-21T08:43:34.100244+0000","last_undegraded":"2024-03-21T08:43:34.100244+0000","last_fullsized":"2024-03-21T08:43:34.100244+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:20:52.859039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094188+0000","last_change":"2024-03-21T08:43:34.089864+0000","last_active":"2024-03-21T08:43:34.094188+0000","last_peered":"2024-03-21T08:43:34.094188+0000","last_clean":"2024-03-21T08:43:34.094188+0000","last_became_active":"2024-03-21T08:43:34.089605+0000","last_became_peered":"2024-03-21T08:43:34.089605+0000","last_unstale":"2024-03-21T08:43:34.094188+0000","last_undegraded":"2024-03-21T08:43:34.094188+0000","last_fullsized":"2024-03-21T08:43:34.094188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:24.367969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074927+0000","last_change":"2024-03-21T08:43:34.070708+0000","last_active":"2024-03-21T08:43:34.074927+0000","last_peered":"2024-03-21T08:43:34.074927+0000","last_clean":"2024-03-21T08:43:34.074927+0000","last_became_active":"2024-03-21T08:43:34.070458+0000","last_became_peered":"2024-03-21T08:43:34.070458+0000","last_unstale":"2024-03-21T08:43:34.074927+0000","last_undegraded":"2024-03-21T08:43:34.074927+0000","last_fullsized":"2024-03-21T08:43:34.074927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:48.402969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069742+0000","last_change":"2024-03-21T08:43:33.069742+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069742+0000","last_undegraded":"2024-03-21T08:43:33.069742+0000","last_fullsized":"2024-03-21T08:43:33.069742+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065512+0000","last_change":"2024-03-21T08:43:33.065512+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065512+0000","last_undegraded":"2024-03-21T08:43:33.065512+0000","last_fullsized":"2024-03-21T08:43:33.065512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080452+0000","last_change":"2024-03-21T08:43:34.074171+0000","last_active":"2024-03-21T08:43:34.080452+0000","last_peered":"2024-03-21T08:43:34.080452+0000","last_clean":"2024-03-21T08:43:34.080452+0000","last_became_active":"2024-03-21T08:43:34.073918+0000","last_became_peered":"2024-03-21T08:43:34.073918+0000","last_unstale":"2024-03-21T08:43:34.080452+0000","last_undegraded":"2024-03-21T08:43:34.080452+0000","last_fullsized":"2024-03-21T08:43:34.080452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:23:54.394841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068308+0000","last_change":"2024-03-21T08:43:33.068308+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068308+0000","last_undegraded":"2024-03-21T08:43:33.068308+0000","last_fullsized":"2024-03-21T08:43:33.068308+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.085472+0000","last_change":"2024-03-21T08:43:34.079738+0000","last_active":"2024-03-21T08:43:34.085472+0000","last_peered":"2024-03-21T08:43:34.085472+0000","last_clean":"2024-03-21T08:43:34.085472+0000","last_became_active":"2024-03-21T08:43:34.079479+0000","last_became_peered":"2024-03-21T08:43:34.079479+0000","last_unstale":"2024-03-21T08:43:34.085472+0000","last_undegraded":"2024-03-21T08:43:34.085472+0000","last_fullsized":"2024-03-21T08:43:34.085472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:22:25.192237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073626+0000","last_change":"2024-03-21T08:43:34.069752+0000","last_active":"2024-03-21T08:43:34.073626+0000","last_peered":"2024-03-21T08:43:34.073626+0000","last_clean":"2024-03-21T08:43:34.073626+0000","last_became_active":"2024-03-21T08:43:34.068765+0000","last_became_peered":"2024-03-21T08:43:34.068765+0000","last_unstale":"2024-03-21T08:43:34.073626+0000","last_undegraded":"2024-03-21T08:43:34.073626+0000","last_fullsized":"2024-03-21T08:43:34.073626+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:28.749067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063487+0000","last_change":"2024-03-21T08:43:33.063487+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063487+0000","last_undegraded":"2024-03-21T08:43:33.063487+0000","last_fullsized":"2024-03-21T08:43:33.063487+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064212+0000","last_change":"2024-03-21T08:43:33.064212+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064212+0000","last_undegraded":"2024-03-21T08:43:33.064212+0000","last_fullsized":"2024-03-21T08:43:33.064212+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084794+0000","last_change":"2024-03-21T08:43:33.081655+0000","last_active":"2024-03-21T08:43:33.084794+0000","last_peered":"2024-03-21T08:43:33.084794+0000","last_clean":"2024-03-21T08:43:33.084794+0000","last_became_active":"2024-03-21T08:43:33.081260+0000","last_became_peered":"2024-03-21T08:43:33.081260+0000","last_unstale":"2024-03-21T08:43:33.084794+0000","last_undegraded":"2024-03-21T08:43:33.084794+0000","last_fullsized":"2024-03-21T08:43:33.084794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074406+0000","last_change":"2024-03-21T08:43:34.070772+0000","last_active":"2024-03-21T08:43:34.074406+0000","last_peered":"2024-03-21T08:43:34.074406+0000","last_clean":"2024-03-21T08:43:34.074406+0000","last_became_active":"2024-03-21T08:43:34.070484+0000","last_became_peered":"2024-03-21T08:43:34.070484+0000","last_unstale":"2024-03-21T08:43:34.074406+0000","last_undegraded":"2024-03-21T08:43:34.074406+0000","last_fullsized":"2024-03-21T08:43:34.074406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084121+0000","last_change":"2024-03-21T08:43:33.078482+0000","last_active":"2024-03-21T08:43:33.084121+0000","last_peered":"2024-03-21T08:43:33.084121+0000","last_clean":"2024-03-21T08:43:33.084121+0000","last_became_active":"2024-03-21T08:43:33.078221+0000","last_became_peered":"2024-03-21T08:43:33.078221+0000","last_unstale":"2024-03-21T08:43:33.084121+0000","last_undegraded":"2024-03-21T08:43:33.084121+0000","last_fullsized":"2024-03-21T08:43:33.084121+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062843+0000","last_change":"2024-03-21T08:43:33.062843+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062843+0000","last_undegraded":"2024-03-21T08:43:33.062843+0000","last_fullsized":"2024-03-21T08:43:33.062843+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084733+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084733+0000","last_peered":"2024-03-21T08:43:33.084733+0000","last_clean":"2024-03-21T08:43:33.084733+0000","last_became_active":"2024-03-21T08:43:33.080225+0000","last_became_peered":"2024-03-21T08:43:33.080225+0000","last_unstale":"2024-03-21T08:43:33.084733+0000","last_undegraded":"2024-03-21T08:43:33.084733+0000","last_fullsized":"2024-03-21T08:43:33.084733+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065020+0000","last_change":"2024-03-21T08:43:33.065020+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065020+0000","last_undegraded":"2024-03-21T08:43:33.065020+0000","last_fullsized":"2024-03-21T08:43:33.065020+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.086635+0000","last_change":"2024-03-21T08:43:33.083210+0000","last_active":"2024-03-21T08:43:33.086635+0000","last_peered":"2024-03-21T08:43:33.086635+0000","last_clean":"2024-03-21T08:43:33.086635+0000","last_became_active":"2024-03-21T08:43:33.082822+0000","last_became_peered":"2024-03-21T08:43:33.082822+0000","last_unstale":"2024-03-21T08:43:33.086635+0000","last_undegraded":"2024-03-21T08:43:33.086635+0000","last_fullsized":"2024-03-21T08:43:33.086635+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064047+0000","last_change":"2024-03-21T08:43:33.064047+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064047+0000","last_undegraded":"2024-03-21T08:43:33.064047+0000","last_fullsized":"2024-03-21T08:43:33.064047+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084144+0000","last_change":"2024-03-21T08:43:33.078204+0000","last_active":"2024-03-21T08:43:33.084144+0000","last_peered":"2024-03-21T08:43:33.084144+0000","last_clean":"2024-03-21T08:43:33.084144+0000","last_became_active":"2024-03-21T08:43:33.077529+0000","last_became_peered":"2024-03-21T08:43:33.077529+0000","last_unstale":"2024-03-21T08:43:33.084144+0000","last_undegraded":"2024-03-21T08:43:33.084144+0000","last_fullsized":"2024-03-21T08:43:33.084144+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:35:40.609258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069118+0000","last_change":"2024-03-21T08:43:33.069118+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069118+0000","last_undegraded":"2024-03-21T08:43:33.069118+0000","last_fullsized":"2024-03-21T08:43:33.069118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084242+0000","last_change":"2024-03-21T08:43:33.078883+0000","last_active":"2024-03-21T08:43:33.084242+0000","last_peered":"2024-03-21T08:43:33.084242+0000","last_clean":"2024-03-21T08:43:33.084242+0000","last_became_active":"2024-03-21T08:43:33.078618+0000","last_became_peered":"2024-03-21T08:43:33.078618+0000","last_unstale":"2024-03-21T08:43:33.084242+0000","last_undegraded":"2024-03-21T08:43:33.084242+0000","last_fullsized":"2024-03-21T08:43:33.084242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084441+0000","last_change":"2024-03-21T08:43:34.082546+0000","last_active":"2024-03-21T08:43:34.084441+0000","last_peered":"2024-03-21T08:43:34.084441+0000","last_clean":"2024-03-21T08:43:34.084441+0000","last_became_active":"2024-03-21T08:43:34.082250+0000","last_became_peered":"2024-03-21T08:43:34.082250+0000","last_unstale":"2024-03-21T08:43:34.084441+0000","last_undegraded":"2024-03-21T08:43:34.084441+0000","last_fullsized":"2024-03-21T08:43:34.084441+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:35.232127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081037+0000","last_change":"2024-03-21T08:43:33.075815+0000","last_active":"2024-03-21T08:43:33.081037+0000","last_peered":"2024-03-21T08:43:33.081037+0000","last_clean":"2024-03-21T08:43:33.081037+0000","last_became_active":"2024-03-21T08:43:33.075533+0000","last_became_peered":"2024-03-21T08:43:33.075533+0000","last_unstale":"2024-03-21T08:43:33.081037+0000","last_undegraded":"2024-03-21T08:43:33.081037+0000","last_fullsized":"2024-03-21T08:43:33.081037+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063254+0000","last_change":"2024-03-21T08:43:33.063254+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063254+0000","last_undegraded":"2024-03-21T08:43:33.063254+0000","last_fullsized":"2024-03-21T08:43:33.063254+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081384+0000","last_change":"2024-03-21T08:43:33.076297+0000","last_active":"2024-03-21T08:43:33.081384+0000","last_peered":"2024-03-21T08:43:33.081384+0000","last_clean":"2024-03-21T08:43:33.081384+0000","last_became_active":"2024-03-21T08:43:33.075418+0000","last_became_peered":"2024-03-21T08:43:33.075418+0000","last_unstale":"2024-03-21T08:43:33.081384+0000","last_undegraded":"2024-03-21T08:43:33.081384+0000","last_fullsized":"2024-03-21T08:43:33.081384+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:12:14.723224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063758+0000","last_change":"2024-03-21T08:43:33.063758+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063758+0000","last_undegraded":"2024-03-21T08:43:33.063758+0000","last_fullsized":"2024-03-21T08:43:33.063758+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.066551+0000","last_change":"2024-03-21T08:43:33.084904+0000","last_active":"2024-03-21T08:43:34.066551+0000","last_peered":"2024-03-21T08:43:34.066551+0000","last_clean":"2024-03-21T08:43:34.066551+0000","last_became_active":"2024-03-21T08:43:33.084490+0000","last_became_peered":"2024-03-21T08:43:33.084490+0000","last_unstale":"2024-03-21T08:43:34.066551+0000","last_undegraded":"2024-03-21T08:43:34.066551+0000","last_fullsized":"2024-03-21T08:43:34.066551+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:59.934028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080558+0000","last_change":"2024-03-21T08:43:34.073629+0000","last_active":"2024-03-21T08:43:34.080558+0000","last_peered":"2024-03-21T08:43:34.080558+0000","last_clean":"2024-03-21T08:43:34.080558+0000","last_became_active":"2024-03-21T08:43:34.073264+0000","last_became_peered":"2024-03-21T08:43:34.073264+0000","last_unstale":"2024-03-21T08:43:34.080558+0000","last_undegraded":"2024-03-21T08:43:34.080558+0000","last_fullsized":"2024-03-21T08:43:34.080558+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:03:33.956736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084141+0000","last_change":"2024-03-21T08:43:33.080472+0000","last_active":"2024-03-21T08:43:33.084141+0000","last_peered":"2024-03-21T08:43:33.084141+0000","last_clean":"2024-03-21T08:43:33.084141+0000","last_became_active":"2024-03-21T08:43:33.080295+0000","last_became_peered":"2024-03-21T08:43:33.080295+0000","last_unstale":"2024-03-21T08:43:33.084141+0000","last_undegraded":"2024-03-21T08:43:33.084141+0000","last_fullsized":"2024-03-21T08:43:33.084141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084312+0000","last_change":"2024-03-21T08:43:34.082659+0000","last_active":"2024-03-21T08:43:34.084312+0000","last_peered":"2024-03-21T08:43:34.084312+0000","last_clean":"2024-03-21T08:43:34.084312+0000","last_became_active":"2024-03-21T08:43:34.082297+0000","last_became_peered":"2024-03-21T08:43:34.082297+0000","last_unstale":"2024-03-21T08:43:34.084312+0000","last_undegraded":"2024-03-21T08:43:34.084312+0000","last_fullsized":"2024-03-21T08:43:34.084312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:58:25.147246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"53'3","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100215+0000","last_change":"2024-03-21T08:43:33.079062+0000","last_active":"2024-03-21T08:43:34.100215+0000","last_peered":"2024-03-21T08:43:34.100215+0000","last_clean":"2024-03-21T08:43:34.100215+0000","last_became_active":"2024-03-21T08:43:33.078759+0000","last_became_peered":"2024-03-21T08:43:33.078759+0000","last_unstale":"2024-03-21T08:43:34.100215+0000","last_undegraded":"2024-03-21T08:43:34.100215+0000","last_fullsized":"2024-03-21T08:43:34.100215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:09.919722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079916+0000","last_change":"2024-03-21T08:43:34.074492+0000","last_active":"2024-03-21T08:43:34.079916+0000","last_peered":"2024-03-21T08:43:34.079916+0000","last_clean":"2024-03-21T08:43:34.079916+0000","last_became_active":"2024-03-21T08:43:34.074094+0000","last_became_peered":"2024-03-21T08:43:34.074094+0000","last_unstale":"2024-03-21T08:43:34.079916+0000","last_undegraded":"2024-03-21T08:43:34.079916+0000","last_fullsized":"2024-03-21T08:43:34.079916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:53:30.468809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081345+0000","last_change":"2024-03-21T08:43:33.074842+0000","last_active":"2024-03-21T08:43:33.081345+0000","last_peered":"2024-03-21T08:43:33.081345+0000","last_clean":"2024-03-21T08:43:33.081345+0000","last_became_active":"2024-03-21T08:43:33.074382+0000","last_became_peered":"2024-03-21T08:43:33.074382+0000","last_unstale":"2024-03-21T08:43:33.081345+0000","last_undegraded":"2024-03-21T08:43:33.081345+0000","last_fullsized":"2024-03-21T08:43:33.081345+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:20:56.748372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068759+0000","last_change":"2024-03-21T08:43:33.068759+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068759+0000","last_undegraded":"2024-03-21T08:43:33.068759+0000","last_fullsized":"2024-03-21T08:43:33.068759+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084080+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084080+0000","last_peered":"2024-03-21T08:43:33.084080+0000","last_clean":"2024-03-21T08:43:33.084080+0000","last_became_active":"2024-03-21T08:43:33.080224+0000","last_became_peered":"2024-03-21T08:43:33.080224+0000","last_unstale":"2024-03-21T08:43:33.084080+0000","last_undegraded":"2024-03-21T08:43:33.084080+0000","last_fullsized":"2024-03-21T08:43:33.084080+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080983+0000","last_change":"2024-03-21T08:43:34.078879+0000","last_active":"2024-03-21T08:43:34.080983+0000","last_peered":"2024-03-21T08:43:34.080983+0000","last_clean":"2024-03-21T08:43:34.080983+0000","last_became_active":"2024-03-21T08:43:34.078610+0000","last_became_peered":"2024-03-21T08:43:34.078610+0000","last_unstale":"2024-03-21T08:43:34.080983+0000","last_undegraded":"2024-03-21T08:43:34.080983+0000","last_fullsized":"2024-03-21T08:43:34.080983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:32.786072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072634+0000","last_change":"2024-03-21T08:43:33.085060+0000","last_active":"2024-03-21T08:43:34.072634+0000","last_peered":"2024-03-21T08:43:34.072634+0000","last_clean":"2024-03-21T08:43:34.072634+0000","last_became_active":"2024-03-21T08:43:33.084725+0000","last_became_peered":"2024-03-21T08:43:33.084725+0000","last_unstale":"2024-03-21T08:43:34.072634+0000","last_undegraded":"2024-03-21T08:43:34.072634+0000","last_fullsized":"2024-03-21T08:43:34.072634+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063545+0000","last_change":"2024-03-21T08:43:33.063545+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063545+0000","last_undegraded":"2024-03-21T08:43:33.063545+0000","last_fullsized":"2024-03-21T08:43:33.063545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081061+0000","last_change":"2024-03-21T08:43:33.075978+0000","last_active":"2024-03-21T08:43:33.081061+0000","last_peered":"2024-03-21T08:43:33.081061+0000","last_clean":"2024-03-21T08:43:33.081061+0000","last_became_active":"2024-03-21T08:43:33.075230+0000","last_became_peered":"2024-03-21T08:43:33.075230+0000","last_unstale":"2024-03-21T08:43:33.081061+0000","last_undegraded":"2024-03-21T08:43:33.081061+0000","last_fullsized":"2024-03-21T08:43:33.081061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.088395+0000","last_change":"2024-03-21T08:43:34.081971+0000","last_active":"2024-03-21T08:43:34.088395+0000","last_peered":"2024-03-21T08:43:34.088395+0000","last_clean":"2024-03-21T08:43:34.088395+0000","last_became_active":"2024-03-21T08:43:34.080960+0000","last_became_peered":"2024-03-21T08:43:34.080960+0000","last_unstale":"2024-03-21T08:43:34.088395+0000","last_undegraded":"2024-03-21T08:43:34.088395+0000","last_fullsized":"2024-03-21T08:43:34.088395+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:54.133660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"53'8","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.098747+0000","last_change":"2024-03-21T08:43:33.082199+0000","last_active":"2024-03-21T08:43:34.098747+0000","last_peered":"2024-03-21T08:43:34.098747+0000","last_clean":"2024-03-21T08:43:34.098747+0000","last_became_active":"2024-03-21T08:43:33.081892+0000","last_became_peered":"2024-03-21T08:43:33.081892+0000","last_unstale":"2024-03-21T08:43:34.098747+0000","last_undegraded":"2024-03-21T08:43:34.098747+0000","last_fullsized":"2024-03-21T08:43:34.098747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:19.722045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.092883+0000","last_change":"2024-03-21T08:43:34.086279+0000","last_active":"2024-03-21T08:43:34.092883+0000","last_peered":"2024-03-21T08:43:34.092883+0000","last_clean":"2024-03-21T08:43:34.092883+0000","last_became_active":"2024-03-21T08:43:34.086021+0000","last_became_peered":"2024-03-21T08:43:34.086021+0000","last_unstale":"2024-03-21T08:43:34.092883+0000","last_undegraded":"2024-03-21T08:43:34.092883+0000","last_fullsized":"2024-03-21T08:43:34.092883+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:51.667945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084347+0000","last_change":"2024-03-21T08:43:33.082173+0000","last_active":"2024-03-21T08:43:33.084347+0000","last_peered":"2024-03-21T08:43:33.084347+0000","last_clean":"2024-03-21T08:43:33.084347+0000","last_became_active":"2024-03-21T08:43:33.081902+0000","last_became_peered":"2024-03-21T08:43:33.081902+0000","last_unstale":"2024-03-21T08:43:33.084347+0000","last_undegraded":"2024-03-21T08:43:33.084347+0000","last_fullsized":"2024-03-21T08:43:33.084347+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065237+0000","last_change":"2024-03-21T08:43:33.065237+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065237+0000","last_undegraded":"2024-03-21T08:43:33.065237+0000","last_fullsized":"2024-03-21T08:43:33.065237+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083890+0000","last_change":"2024-03-21T08:43:33.077524+0000","last_active":"2024-03-21T08:43:33.083890+0000","last_peered":"2024-03-21T08:43:33.083890+0000","last_clean":"2024-03-21T08:43:33.083890+0000","last_became_active":"2024-03-21T08:43:33.077163+0000","last_became_peered":"2024-03-21T08:43:33.077163+0000","last_unstale":"2024-03-21T08:43:33.083890+0000","last_undegraded":"2024-03-21T08:43:33.083890+0000","last_fullsized":"2024-03-21T08:43:33.083890+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069726+0000","last_change":"2024-03-21T08:43:33.069726+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069726+0000","last_undegraded":"2024-03-21T08:43:33.069726+0000","last_fullsized":"2024-03-21T08:43:33.069726+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083836+0000","last_change":"2024-03-21T08:43:33.078372+0000","last_active":"2024-03-21T08:43:33.083836+0000","last_peered":"2024-03-21T08:43:33.083836+0000","last_clean":"2024-03-21T08:43:33.083836+0000","last_became_active":"2024-03-21T08:43:33.077797+0000","last_became_peered":"2024-03-21T08:43:33.077797+0000","last_unstale":"2024-03-21T08:43:33.083836+0000","last_undegraded":"2024-03-21T08:43:33.083836+0000","last_fullsized":"2024-03-21T08:43:33.083836+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.076176+0000","last_change":"2024-03-21T08:43:34.072036+0000","last_active":"2024-03-21T08:43:34.076176+0000","last_peered":"2024-03-21T08:43:34.076176+0000","last_clean":"2024-03-21T08:43:34.076176+0000","last_became_active":"2024-03-21T08:43:34.071710+0000","last_became_peered":"2024-03-21T08:43:34.071710+0000","last_unstale":"2024-03-21T08:43:34.076176+0000","last_undegraded":"2024-03-21T08:43:34.076176+0000","last_fullsized":"2024-03-21T08:43:34.076176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:37.084279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084283+0000","last_change":"2024-03-21T08:43:33.079120+0000","last_active":"2024-03-21T08:43:33.084283+0000","last_peered":"2024-03-21T08:43:33.084283+0000","last_clean":"2024-03-21T08:43:33.084283+0000","last_became_active":"2024-03-21T08:43:33.078672+0000","last_became_peered":"2024-03-21T08:43:33.078672+0000","last_unstale":"2024-03-21T08:43:33.084283+0000","last_undegraded":"2024-03-21T08:43:33.084283+0000","last_fullsized":"2024-03-21T08:43:33.084283+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068324+0000","last_change":"2024-03-21T08:43:33.068324+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068324+0000","last_undegraded":"2024-03-21T08:43:33.068324+0000","last_fullsized":"2024-03-21T08:43:33.068324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076791+0000","last_change":"2024-03-21T08:43:33.074311+0000","last_active":"2024-03-21T08:43:33.076791+0000","last_peered":"2024-03-21T08:43:33.076791+0000","last_clean":"2024-03-21T08:43:33.076791+0000","last_became_active":"2024-03-21T08:43:33.074047+0000","last_became_peered":"2024-03-21T08:43:33.074047+0000","last_unstale":"2024-03-21T08:43:33.076791+0000","last_undegraded":"2024-03-21T08:43:33.076791+0000","last_fullsized":"2024-03-21T08:43:33.076791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062142+0000","last_change":"2024-03-21T08:43:33.062142+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062142+0000","last_undegraded":"2024-03-21T08:43:33.062142+0000","last_fullsized":"2024-03-21T08:43:33.062142+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.062005+0000","last_change":"2024-03-21T08:43:33.080008+0000","last_active":"2024-03-21T08:43:34.062005+0000","last_peered":"2024-03-21T08:43:34.062005+0000","last_clean":"2024-03-21T08:43:34.062005+0000","last_became_active":"2024-03-21T08:43:33.079744+0000","last_became_peered":"2024-03-21T08:43:33.079744+0000","last_unstale":"2024-03-21T08:43:34.062005+0000","last_undegraded":"2024-03-21T08:43:34.062005+0000","last_fullsized":"2024-03-21T08:43:34.062005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:34.225900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062049+0000","last_change":"2024-03-21T08:43:33.062049+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062049+0000","last_undegraded":"2024-03-21T08:43:33.062049+0000","last_fullsized":"2024-03-21T08:43:33.062049+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081118+0000","last_change":"2024-03-21T08:43:33.076251+0000","last_active":"2024-03-21T08:43:33.081118+0000","last_peered":"2024-03-21T08:43:33.081118+0000","last_clean":"2024-03-21T08:43:33.081118+0000","last_became_active":"2024-03-21T08:43:33.074661+0000","last_became_peered":"2024-03-21T08:43:33.074661+0000","last_unstale":"2024-03-21T08:43:33.081118+0000","last_undegraded":"2024-03-21T08:43:33.081118+0000","last_fullsized":"2024-03-21T08:43:33.081118+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064097+0000","last_change":"2024-03-21T08:43:33.064097+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064097+0000","last_undegraded":"2024-03-21T08:43:33.064097+0000","last_fullsized":"2024-03-21T08:43:33.064097+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"53'2","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.106185+0000","last_change":"2024-03-21T08:43:33.079309+0000","last_active":"2024-03-21T08:43:34.106185+0000","last_peered":"2024-03-21T08:43:34.106185+0000","last_clean":"2024-03-21T08:43:34.106185+0000","last_became_active":"2024-03-21T08:43:33.079038+0000","last_became_peered":"2024-03-21T08:43:33.079038+0000","last_unstale":"2024-03-21T08:43:34.106185+0000","last_undegraded":"2024-03-21T08:43:34.106185+0000","last_fullsized":"2024-03-21T08:43:34.106185+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:23.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064702+0000","last_change":"2024-03-21T08:43:33.064702+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064702+0000","last_undegraded":"2024-03-21T08:43:33.064702+0000","last_fullsized":"2024-03-21T08:43:33.064702+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084195+0000","last_change":"2024-03-21T08:43:33.078636+0000","last_active":"2024-03-21T08:43:33.084195+0000","last_peered":"2024-03-21T08:43:33.084195+0000","last_clean":"2024-03-21T08:43:33.084195+0000","last_became_active":"2024-03-21T08:43:33.078336+0000","last_became_peered":"2024-03-21T08:43:33.078336+0000","last_unstale":"2024-03-21T08:43:33.084195+0000","last_undegraded":"2024-03-21T08:43:33.084195+0000","last_fullsized":"2024-03-21T08:43:33.084195+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084289+0000","last_change":"2024-03-21T08:43:34.082469+0000","last_active":"2024-03-21T08:43:34.084289+0000","last_peered":"2024-03-21T08:43:34.084289+0000","last_clean":"2024-03-21T08:43:34.084289+0000","last_became_active":"2024-03-21T08:43:34.082227+0000","last_became_peered":"2024-03-21T08:43:34.082227+0000","last_unstale":"2024-03-21T08:43:34.084289+0000","last_undegraded":"2024-03-21T08:43:34.084289+0000","last_fullsized":"2024-03-21T08:43:34.084289+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:14.199100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.071019+0000","last_change":"2024-03-21T08:43:33.077725+0000","last_active":"2024-03-21T08:43:34.071019+0000","last_peered":"2024-03-21T08:43:34.071019+0000","last_clean":"2024-03-21T08:43:34.071019+0000","last_became_active":"2024-03-21T08:43:33.077452+0000","last_became_peered":"2024-03-21T08:43:33.077452+0000","last_unstale":"2024-03-21T08:43:34.071019+0000","last_undegraded":"2024-03-21T08:43:34.071019+0000","last_fullsized":"2024-03-21T08:43:34.071019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:22:33.956371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064263+0000","last_change":"2024-03-21T08:43:33.064263+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064263+0000","last_undegraded":"2024-03-21T08:43:33.064263+0000","last_fullsized":"2024-03-21T08:43:33.064263+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081515+0000","last_change":"2024-03-21T08:43:33.076652+0000","last_active":"2024-03-21T08:43:33.081515+0000","last_peered":"2024-03-21T08:43:33.081515+0000","last_clean":"2024-03-21T08:43:33.081515+0000","last_became_active":"2024-03-21T08:43:33.074101+0000","last_became_peered":"2024-03-21T08:43:33.074101+0000","last_unstale":"2024-03-21T08:43:33.081515+0000","last_undegraded":"2024-03-21T08:43:33.081515+0000","last_fullsized":"2024-03-21T08:43:33.081515+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063671+0000","last_change":"2024-03-21T08:43:33.063671+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063671+0000","last_undegraded":"2024-03-21T08:43:33.063671+0000","last_fullsized":"2024-03-21T08:43:33.063671+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061812+0000","last_change":"2024-03-21T08:43:33.081506+0000","last_active":"2024-03-21T08:43:34.061812+0000","last_peered":"2024-03-21T08:43:34.061812+0000","last_clean":"2024-03-21T08:43:34.061812+0000","last_became_active":"2024-03-21T08:43:33.080894+0000","last_became_peered":"2024-03-21T08:43:33.080894+0000","last_unstale":"2024-03-21T08:43:34.061812+0000","last_undegraded":"2024-03-21T08:43:34.061812+0000","last_fullsized":"2024-03-21T08:43:34.061812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093878+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:34.093878+0000","last_peered":"2024-03-21T08:43:34.093878+0000","last_clean":"2024-03-21T08:43:34.093878+0000","last_became_active":"2024-03-21T08:43:34.089436+0000","last_became_peered":"2024-03-21T08:43:34.089436+0000","last_unstale":"2024-03-21T08:43:34.093878+0000","last_undegraded":"2024-03-21T08:43:34.093878+0000","last_fullsized":"2024-03-21T08:43:34.093878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:10.546830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084246+0000","last_change":"2024-03-21T08:43:33.080132+0000","last_active":"2024-03-21T08:43:33.084246+0000","last_peered":"2024-03-21T08:43:33.084246+0000","last_clean":"2024-03-21T08:43:33.084246+0000","last_became_active":"2024-03-21T08:43:33.079867+0000","last_became_peered":"2024-03-21T08:43:33.079867+0000","last_unstale":"2024-03-21T08:43:33.084246+0000","last_undegraded":"2024-03-21T08:43:33.084246+0000","last_fullsized":"2024-03-21T08:43:33.084246+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:33:42.756681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.089682+0000","last_change":"2024-03-21T08:43:34.084715+0000","last_active":"2024-03-21T08:43:34.089682+0000","last_peered":"2024-03-21T08:43:34.089682+0000","last_clean":"2024-03-21T08:43:34.089682+0000","last_became_active":"2024-03-21T08:43:34.084436+0000","last_became_peered":"2024-03-21T08:43:34.084436+0000","last_unstale":"2024-03-21T08:43:34.089682+0000","last_undegraded":"2024-03-21T08:43:34.089682+0000","last_fullsized":"2024-03-21T08:43:34.089682+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:33:01.770928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061783+0000","last_change":"2024-03-21T08:43:33.076471+0000","last_active":"2024-03-21T08:43:34.061783+0000","last_peered":"2024-03-21T08:43:34.061783+0000","last_clean":"2024-03-21T08:43:34.061783+0000","last_became_active":"2024-03-21T08:43:33.076185+0000","last_became_peered":"2024-03-21T08:43:33.076185+0000","last_unstale":"2024-03-21T08:43:34.061783+0000","last_undegraded":"2024-03-21T08:43:34.061783+0000","last_fullsized":"2024-03-21T08:43:34.061783+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062860+0000","last_change":"2024-03-21T08:43:33.062860+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062860+0000","last_undegraded":"2024-03-21T08:43:33.062860+0000","last_fullsized":"2024-03-21T08:43:33.062860+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076315+0000","last_change":"2024-03-21T08:43:33.074819+0000","last_active":"2024-03-21T08:43:33.076315+0000","last_peered":"2024-03-21T08:43:33.076315+0000","last_clean":"2024-03-21T08:43:33.076315+0000","last_became_active":"2024-03-21T08:43:33.074585+0000","last_became_peered":"2024-03-21T08:43:33.074585+0000","last_unstale":"2024-03-21T08:43:33.076315+0000","last_undegraded":"2024-03-21T08:43:33.076315+0000","last_fullsized":"2024-03-21T08:43:33.076315+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063064+0000","last_change":"2024-03-21T08:43:33.063064+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063064+0000","last_undegraded":"2024-03-21T08:43:33.063064+0000","last_fullsized":"2024-03-21T08:43:33.063064+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072378+0000","last_change":"2024-03-21T08:43:33.081102+0000","last_active":"2024-03-21T08:43:34.072378+0000","last_peered":"2024-03-21T08:43:34.072378+0000","last_clean":"2024-03-21T08:43:34.072378+0000","last_became_active":"2024-03-21T08:43:33.080820+0000","last_became_peered":"2024-03-21T08:43:33.080820+0000","last_unstale":"2024-03-21T08:43:34.072378+0000","last_undegraded":"2024-03-21T08:43:34.072378+0000","last_fullsized":"2024-03-21T08:43:34.072378+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.065013+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:43:34.065013+0000","last_peered":"2024-03-21T08:43:34.065013+0000","last_clean":"2024-03-21T08:43:34.065013+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:43:34.065013+0000","last_undegraded":"2024-03-21T08:43:34.065013+0000","last_fullsized":"2024-03-21T08:43:34.065013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509476,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1228800,"data_stored":871737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":281209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":281299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":602112,"data_stored":267383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":864410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:43:38.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:43:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.362+0000 7f1d31400640 1 Processor -- start 2024-03-21T08:43:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.362+0000 7f1d31400640 1 -- start start 2024-03-21T08:43:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d31400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d2c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d31400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0cdb50 con 0x7f1d2c09a290 2024-03-21T08:43:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d31400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0cde30 con 0x7f1d2c0cd5d0 2024-03-21T08:43:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d31400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0c5530 con 0x7f1d2c15f260 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2b200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1d2c09a290 0x7f1d2c0c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38684/0 (socket says 172.21.15.28:38684) 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2b200640 1 -- 172.21.15.28:0/1754173558 learned_addr learned my addr 172.21.15.28:0/1754173558 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d2c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071334899 0 0) 0x7f1d2c0cdb50 con 0x7f1d2c09a290 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d10003530 con 0x7f1d2c09a290 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346484537 0 0) 0x7f1d2c0cde30 con 0x7f1d2c0cd5d0 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c0cdb50 con 0x7f1d2c0cd5d0 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4130453489 0 0) 0x7f1d10003530 con 0x7f1d2c09a290 2024-03-21T08:43:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 msgr2=0x7f1d2c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d2c0cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 >> v1:172.21.15.151:6789/0 conn(0x7f1d2c0cd5d0 legacy=0x7f1d2c099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0c5780 con 0x7f1d2c09a290 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f1d24001980 con 0x7f1d2c09a290 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 >> v1:172.21.15.28:6789/0 conn(0x7f1d2c09a290 legacy=0x7f1d2c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10003b80 0x7f1d10003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d10004810 0x7f1d10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0c5530 con 0x7f1d10003b80 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1000bad0 con 0x7f1d10004810 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1000bcd0 con 0x7f1d2c15f260 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10003b80 0x7f1d10003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2b200640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d21e00640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d10004810 0x7f1d10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 -- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 msgr2=0x7f1d100051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d2c15f260 0x7f1d100051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 -- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d10004810 msgr2=0x7f1d10004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d10004810 0x7f1d10004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.366+0000 7f1d2a800640 1 -- 172.21.15.28:0/1754173558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10003530 con 0x7f1d10003b80 2024-03-21T08:43:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.370+0000 7f1d2a800640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10003b80 0x7f1d10003f60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009ae0 tx=0x7f1d1c01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a6749a5bca2090 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.370+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0096b0 con 0x7f1d10003b80 2024-03-21T08:43:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.370+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c0a6450 con 0x7f1d10003b80 2024-03-21T08:43:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.370+0000 7f1d29e00640 1 -- 172.21.15.28:0/1754173558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0a6840 con 0x7f1d10003b80 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- 172.21.15.28:0/1754173558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10003b80 msgr2=0x7f1d10003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 --2- 172.21.15.28:0/1754173558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10003b80 0x7f1d10003f60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009ae0 tx=0x7f1d1c01ca40 comp rx=0 tx=0).stop 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d2b200640 1 -- 172.21.15.28:0/1754173558 reap_dead start 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- 172.21.15.28:0/1754173558 shutdown_connections 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- 172.21.15.28:0/1754173558 >> 172.21.15.28:0/1754173558 conn(0x7f1d2c0b34d0 msgr2=0x7f1d2c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- 172.21.15.28:0/1754173558 shutdown_connections 2024-03-21T08:43:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- 172.21.15.28:0/1754173558 wait complete. 2024-03-21T08:43:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 Processor -- start 2024-03-21T08:43:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.374+0000 7f1d31400640 1 -- start start 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d10003b80 0x7f1d2c0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 0x7f1d2c1f9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0c7020 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0c6ea0 con 0x7f1d2c06d810 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0c71a0 con 0x7f1d10003b80 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43470/0 (socket says 172.21.15.28:43470) 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 -- 172.21.15.28:0/4224931772 learned_addr learned my addr 172.21.15.28:0/4224931772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d2a800640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d10003b80 0x7f1d2c0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d2b200640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 0x7f1d2c1f9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 -- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d10003b80 msgr2=0x7f1d2c0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d10003b80 0x7f1d2c0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 -- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 msgr2=0x7f1d2c1f9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 0x7f1d2c1f9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 -- 172.21.15.28:0/4224931772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0b2380 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d2b200640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 0x7f1d2c1f9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d21e00640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400f870 tx=0x7f1d1400fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1401a070 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d14005320 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d140069e0 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c0ca7f0 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c0c9a00 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.378+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c04f410 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.382+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1d1401d070 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.382+0000 7f1d23400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 0x7f1d0806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.382+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d14085a40 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.382+0000 7f1d2a800640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 0x7f1d0806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.382+0000 7f1d2a800640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 0x7f1d0806a010 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0ad000 tx=0x7f1d1c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.386+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1d140562b0 con 0x7f1d2c15a1a0 2024-03-21T08:43:38.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.546+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d2c06dbf0 con 0x7f1d08067b50 2024-03-21T08:43:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.550+0000 7f1d23400640 1 -- 172.21.15.28:0/4224931772 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+248843 (secure 0 0 0) 0x7f1d2c06dbf0 con 0x7f1d08067b50 2024-03-21T08:43:38.557 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:38.558 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 msgr2=0x7f1d0806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 0x7f1d0806a010 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0ad000 tx=0x7f1d1c0046d0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 msgr2=0x7f1d2c06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400f870 tx=0x7f1d1400fd30 comp rx=0 tx=0).stop 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 shutdown_connections 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1d2c06d810 0x7f1d2c1f9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d2c15a1a0 0x7f1d2c06d2d0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1d08067b50 0x7f1d0806a010 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 --2- 172.21.15.28:0/4224931772 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1d10003b80 0x7f1d2c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 >> 172.21.15.28:0/4224931772 conn(0x7f1d2c0b34d0 msgr2=0x7f1d2c099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:38.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 shutdown_connections 2024-03-21T08:43:38.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.554+0000 7f1d31400640 1 -- 172.21.15.28:0/4224931772 wait complete. 2024-03-21T08:43:38.586 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-21T08:43:37.753502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":178,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238812,"kb_used_data":5996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735908,"statfs":{"total":773094113280,"available":772849569792,"internally_reserved":0,"allocated":6139904,"data_stored":3388164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705743"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.105337+0000","last_change":"2024-03-21T08:43:34.104498+0000","last_active":"2024-03-21T08:43:34.105337+0000","last_peered":"2024-03-21T08:43:34.105337+0000","last_clean":"2024-03-21T08:43:34.105337+0000","last_became_active":"2024-03-21T08:43:34.102336+0000","last_became_peered":"2024-03-21T08:43:34.102336+0000","last_unstale":"2024-03-21T08:43:34.105337+0000","last_undegraded":"2024-03-21T08:43:34.105337+0000","last_fullsized":"2024-03-21T08:43:34.105337+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:44:57.769559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.066283+0000","last_change":"2024-03-21T08:43:33.066283+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.066283+0000","last_undegraded":"2024-03-21T08:43:33.066283+0000","last_fullsized":"2024-03-21T08:43:33.066283+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084512+0000","last_change":"2024-03-21T08:43:34.079798+0000","last_active":"2024-03-21T08:43:34.084512+0000","last_peered":"2024-03-21T08:43:34.084512+0000","last_clean":"2024-03-21T08:43:34.084512+0000","last_became_active":"2024-03-21T08:43:34.079557+0000","last_became_peered":"2024-03-21T08:43:34.079557+0000","last_unstale":"2024-03-21T08:43:34.084512+0000","last_undegraded":"2024-03-21T08:43:34.084512+0000","last_fullsized":"2024-03-21T08:43:34.084512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:33:45.824524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065481+0000","last_change":"2024-03-21T08:43:33.065481+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065481+0000","last_undegraded":"2024-03-21T08:43:33.065481+0000","last_fullsized":"2024-03-21T08:43:33.065481+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074891+0000","last_change":"2024-03-21T08:43:34.069862+0000","last_active":"2024-03-21T08:43:34.074891+0000","last_peered":"2024-03-21T08:43:34.074891+0000","last_clean":"2024-03-21T08:43:34.074891+0000","last_became_active":"2024-03-21T08:43:34.069561+0000","last_became_peered":"2024-03-21T08:43:34.069561+0000","last_unstale":"2024-03-21T08:43:34.074891+0000","last_undegraded":"2024-03-21T08:43:34.074891+0000","last_fullsized":"2024-03-21T08:43:34.074891+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:02:24.282774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064693+0000","last_change":"2024-03-21T08:43:33.064693+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064693+0000","last_undegraded":"2024-03-21T08:43:33.064693+0000","last_fullsized":"2024-03-21T08:43:33.064693+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065768+0000","last_change":"2024-03-21T08:43:33.065768+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065768+0000","last_undegraded":"2024-03-21T08:43:33.065768+0000","last_fullsized":"2024-03-21T08:43:33.065768+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062573+0000","last_change":"2024-03-21T08:43:33.062573+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062573+0000","last_undegraded":"2024-03-21T08:43:33.062573+0000","last_fullsized":"2024-03-21T08:43:33.062573+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063862+0000","last_change":"2024-03-21T08:43:33.063862+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063862+0000","last_undegraded":"2024-03-21T08:43:33.063862+0000","last_fullsized":"2024-03-21T08:43:33.063862+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079493+0000","last_change":"2024-03-21T08:43:34.076812+0000","last_active":"2024-03-21T08:43:34.079493+0000","last_peered":"2024-03-21T08:43:34.079493+0000","last_clean":"2024-03-21T08:43:34.079493+0000","last_became_active":"2024-03-21T08:43:34.076556+0000","last_became_peered":"2024-03-21T08:43:34.076556+0000","last_unstale":"2024-03-21T08:43:34.079493+0000","last_undegraded":"2024-03-21T08:43:34.079493+0000","last_fullsized":"2024-03-21T08:43:34.079493+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:17:35.654015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.099722+0000","last_change":"2024-03-21T08:43:34.096375+0000","last_active":"2024-03-21T08:43:34.099722+0000","last_peered":"2024-03-21T08:43:34.099722+0000","last_clean":"2024-03-21T08:43:34.099722+0000","last_became_active":"2024-03-21T08:43:34.095943+0000","last_became_peered":"2024-03-21T08:43:34.095943+0000","last_unstale":"2024-03-21T08:43:34.099722+0000","last_undegraded":"2024-03-21T08:43:34.099722+0000","last_fullsized":"2024-03-21T08:43:34.099722+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:20.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065944+0000","last_change":"2024-03-21T08:43:33.065944+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065944+0000","last_undegraded":"2024-03-21T08:43:33.065944+0000","last_fullsized":"2024-03-21T08:43:33.065944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094467+0000","last_change":"2024-03-21T08:43:34.090738+0000","last_active":"2024-03-21T08:43:34.094467+0000","last_peered":"2024-03-21T08:43:34.094467+0000","last_clean":"2024-03-21T08:43:34.094467+0000","last_became_active":"2024-03-21T08:43:34.090493+0000","last_became_peered":"2024-03-21T08:43:34.090493+0000","last_unstale":"2024-03-21T08:43:34.094467+0000","last_undegraded":"2024-03-21T08:43:34.094467+0000","last_fullsized":"2024-03-21T08:43:34.094467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063655+0000","last_change":"2024-03-21T08:43:33.063655+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063655+0000","last_undegraded":"2024-03-21T08:43:33.063655+0000","last_fullsized":"2024-03-21T08:43:33.063655+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063371+0000","last_change":"2024-03-21T08:43:33.063371+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063371+0000","last_undegraded":"2024-03-21T08:43:33.063371+0000","last_fullsized":"2024-03-21T08:43:33.063371+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065030+0000","last_change":"2024-03-21T08:43:33.065030+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065030+0000","last_undegraded":"2024-03-21T08:43:33.065030+0000","last_fullsized":"2024-03-21T08:43:33.065030+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068546+0000","last_change":"2024-03-21T08:43:33.068546+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068546+0000","last_undegraded":"2024-03-21T08:43:33.068546+0000","last_fullsized":"2024-03-21T08:43:33.068546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093933+0000","last_change":"2024-03-21T08:43:34.090757+0000","last_active":"2024-03-21T08:43:34.093933+0000","last_peered":"2024-03-21T08:43:34.093933+0000","last_clean":"2024-03-21T08:43:34.093933+0000","last_became_active":"2024-03-21T08:43:34.090125+0000","last_became_peered":"2024-03-21T08:43:34.090125+0000","last_unstale":"2024-03-21T08:43:34.093933+0000","last_undegraded":"2024-03-21T08:43:34.093933+0000","last_fullsized":"2024-03-21T08:43:34.093933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:14:11.321572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.103454+0000","last_change":"2024-03-21T08:43:34.099533+0000","last_active":"2024-03-21T08:43:34.103454+0000","last_peered":"2024-03-21T08:43:34.103454+0000","last_clean":"2024-03-21T08:43:34.103454+0000","last_became_active":"2024-03-21T08:43:34.099273+0000","last_became_peered":"2024-03-21T08:43:34.099273+0000","last_unstale":"2024-03-21T08:43:34.103454+0000","last_undegraded":"2024-03-21T08:43:34.103454+0000","last_fullsized":"2024-03-21T08:43:34.103454+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:25:59.522961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.070108+0000","last_change":"2024-03-21T08:43:33.070108+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.070108+0000","last_undegraded":"2024-03-21T08:43:33.070108+0000","last_fullsized":"2024-03-21T08:43:33.070108+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073021+0000","last_change":"2024-03-21T08:43:34.069515+0000","last_active":"2024-03-21T08:43:34.073021+0000","last_peered":"2024-03-21T08:43:34.073021+0000","last_clean":"2024-03-21T08:43:34.073021+0000","last_became_active":"2024-03-21T08:43:34.069239+0000","last_became_peered":"2024-03-21T08:43:34.069239+0000","last_unstale":"2024-03-21T08:43:34.073021+0000","last_undegraded":"2024-03-21T08:43:34.073021+0000","last_fullsized":"2024-03-21T08:43:34.073021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:40:59.946986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064406+0000","last_change":"2024-03-21T08:43:33.064406+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064406+0000","last_undegraded":"2024-03-21T08:43:33.064406+0000","last_fullsized":"2024-03-21T08:43:33.064406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100244+0000","last_change":"2024-03-21T08:43:34.097093+0000","last_active":"2024-03-21T08:43:34.100244+0000","last_peered":"2024-03-21T08:43:34.100244+0000","last_clean":"2024-03-21T08:43:34.100244+0000","last_became_active":"2024-03-21T08:43:34.096448+0000","last_became_peered":"2024-03-21T08:43:34.096448+0000","last_unstale":"2024-03-21T08:43:34.100244+0000","last_undegraded":"2024-03-21T08:43:34.100244+0000","last_fullsized":"2024-03-21T08:43:34.100244+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:20:52.859039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094188+0000","last_change":"2024-03-21T08:43:34.089864+0000","last_active":"2024-03-21T08:43:34.094188+0000","last_peered":"2024-03-21T08:43:34.094188+0000","last_clean":"2024-03-21T08:43:34.094188+0000","last_became_active":"2024-03-21T08:43:34.089605+0000","last_became_peered":"2024-03-21T08:43:34.089605+0000","last_unstale":"2024-03-21T08:43:34.094188+0000","last_undegraded":"2024-03-21T08:43:34.094188+0000","last_fullsized":"2024-03-21T08:43:34.094188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:24.367969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074927+0000","last_change":"2024-03-21T08:43:34.070708+0000","last_active":"2024-03-21T08:43:34.074927+0000","last_peered":"2024-03-21T08:43:34.074927+0000","last_clean":"2024-03-21T08:43:34.074927+0000","last_became_active":"2024-03-21T08:43:34.070458+0000","last_became_peered":"2024-03-21T08:43:34.070458+0000","last_unstale":"2024-03-21T08:43:34.074927+0000","last_undegraded":"2024-03-21T08:43:34.074927+0000","last_fullsized":"2024-03-21T08:43:34.074927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:48.402969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069742+0000","last_change":"2024-03-21T08:43:33.069742+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069742+0000","last_undegraded":"2024-03-21T08:43:33.069742+0000","last_fullsized":"2024-03-21T08:43:33.069742+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065512+0000","last_change":"2024-03-21T08:43:33.065512+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065512+0000","last_undegraded":"2024-03-21T08:43:33.065512+0000","last_fullsized":"2024-03-21T08:43:33.065512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080452+0000","last_change":"2024-03-21T08:43:34.074171+0000","last_active":"2024-03-21T08:43:34.080452+0000","last_peered":"2024-03-21T08:43:34.080452+0000","last_clean":"2024-03-21T08:43:34.080452+0000","last_became_active":"2024-03-21T08:43:34.073918+0000","last_became_peered":"2024-03-21T08:43:34.073918+0000","last_unstale":"2024-03-21T08:43:34.080452+0000","last_undegraded":"2024-03-21T08:43:34.080452+0000","last_fullsized":"2024-03-21T08:43:34.080452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:23:54.394841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068308+0000","last_change":"2024-03-21T08:43:33.068308+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068308+0000","last_undegraded":"2024-03-21T08:43:33.068308+0000","last_fullsized":"2024-03-21T08:43:33.068308+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.085472+0000","last_change":"2024-03-21T08:43:34.079738+0000","last_active":"2024-03-21T08:43:34.085472+0000","last_peered":"2024-03-21T08:43:34.085472+0000","last_clean":"2024-03-21T08:43:34.085472+0000","last_became_active":"2024-03-21T08:43:34.079479+0000","last_became_peered":"2024-03-21T08:43:34.079479+0000","last_unstale":"2024-03-21T08:43:34.085472+0000","last_undegraded":"2024-03-21T08:43:34.085472+0000","last_fullsized":"2024-03-21T08:43:34.085472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:22:25.192237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073626+0000","last_change":"2024-03-21T08:43:34.069752+0000","last_active":"2024-03-21T08:43:34.073626+0000","last_peered":"2024-03-21T08:43:34.073626+0000","last_clean":"2024-03-21T08:43:34.073626+0000","last_became_active":"2024-03-21T08:43:34.068765+0000","last_became_peered":"2024-03-21T08:43:34.068765+0000","last_unstale":"2024-03-21T08:43:34.073626+0000","last_undegraded":"2024-03-21T08:43:34.073626+0000","last_fullsized":"2024-03-21T08:43:34.073626+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:28.749067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063487+0000","last_change":"2024-03-21T08:43:33.063487+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063487+0000","last_undegraded":"2024-03-21T08:43:33.063487+0000","last_fullsized":"2024-03-21T08:43:33.063487+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064212+0000","last_change":"2024-03-21T08:43:33.064212+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064212+0000","last_undegraded":"2024-03-21T08:43:33.064212+0000","last_fullsized":"2024-03-21T08:43:33.064212+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084794+0000","last_change":"2024-03-21T08:43:33.081655+0000","last_active":"2024-03-21T08:43:33.084794+0000","last_peered":"2024-03-21T08:43:33.084794+0000","last_clean":"2024-03-21T08:43:33.084794+0000","last_became_active":"2024-03-21T08:43:33.081260+0000","last_became_peered":"2024-03-21T08:43:33.081260+0000","last_unstale":"2024-03-21T08:43:33.084794+0000","last_undegraded":"2024-03-21T08:43:33.084794+0000","last_fullsized":"2024-03-21T08:43:33.084794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074406+0000","last_change":"2024-03-21T08:43:34.070772+0000","last_active":"2024-03-21T08:43:34.074406+0000","last_peered":"2024-03-21T08:43:34.074406+0000","last_clean":"2024-03-21T08:43:34.074406+0000","last_became_active":"2024-03-21T08:43:34.070484+0000","last_became_peered":"2024-03-21T08:43:34.070484+0000","last_unstale":"2024-03-21T08:43:34.074406+0000","last_undegraded":"2024-03-21T08:43:34.074406+0000","last_fullsized":"2024-03-21T08:43:34.074406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084121+0000","last_change":"2024-03-21T08:43:33.078482+0000","last_active":"2024-03-21T08:43:33.084121+0000","last_peered":"2024-03-21T08:43:33.084121+0000","last_clean":"2024-03-21T08:43:33.084121+0000","last_became_active":"2024-03-21T08:43:33.078221+0000","last_became_peered":"2024-03-21T08:43:33.078221+0000","last_unstale":"2024-03-21T08:43:33.084121+0000","last_undegraded":"2024-03-21T08:43:33.084121+0000","last_fullsized":"2024-03-21T08:43:33.084121+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062843+0000","last_change":"2024-03-21T08:43:33.062843+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062843+0000","last_undegraded":"2024-03-21T08:43:33.062843+0000","last_fullsized":"2024-03-21T08:43:33.062843+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084733+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084733+0000","last_peered":"2024-03-21T08:43:33.084733+0000","last_clean":"2024-03-21T08:43:33.084733+0000","last_became_active":"2024-03-21T08:43:33.080225+0000","last_became_peered":"2024-03-21T08:43:33.080225+0000","last_unstale":"2024-03-21T08:43:33.084733+0000","last_undegraded":"2024-03-21T08:43:33.084733+0000","last_fullsized":"2024-03-21T08:43:33.084733+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065020+0000","last_change":"2024-03-21T08:43:33.065020+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065020+0000","last_undegraded":"2024-03-21T08:43:33.065020+0000","last_fullsized":"2024-03-21T08:43:33.065020+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.086635+0000","last_change":"2024-03-21T08:43:33.083210+0000","last_active":"2024-03-21T08:43:33.086635+0000","last_peered":"2024-03-21T08:43:33.086635+0000","last_clean":"2024-03-21T08:43:33.086635+0000","last_became_active":"2024-03-21T08:43:33.082822+0000","last_became_peered":"2024-03-21T08:43:33.082822+0000","last_unstale":"2024-03-21T08:43:33.086635+0000","last_undegraded":"2024-03-21T08:43:33.086635+0000","last_fullsized":"2024-03-21T08:43:33.086635+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064047+0000","last_change":"2024-03-21T08:43:33.064047+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064047+0000","last_undegraded":"2024-03-21T08:43:33.064047+0000","last_fullsized":"2024-03-21T08:43:33.064047+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084144+0000","last_change":"2024-03-21T08:43:33.078204+0000","last_active":"2024-03-21T08:43:33.084144+0000","last_peered":"2024-03-21T08:43:33.084144+0000","last_clean":"2024-03-21T08:43:33.084144+0000","last_became_active":"2024-03-21T08:43:33.077529+0000","last_became_peered":"2024-03-21T08:43:33.077529+0000","last_unstale":"2024-03-21T08:43:33.084144+0000","last_undegraded":"2024-03-21T08:43:33.084144+0000","last_fullsized":"2024-03-21T08:43:33.084144+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:35:40.609258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069118+0000","last_change":"2024-03-21T08:43:33.069118+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069118+0000","last_undegraded":"2024-03-21T08:43:33.069118+0000","last_fullsized":"2024-03-21T08:43:33.069118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084242+0000","last_change":"2024-03-21T08:43:33.078883+0000","last_active":"2024-03-21T08:43:33.084242+0000","last_peered":"2024-03-21T08:43:33.084242+0000","last_clean":"2024-03-21T08:43:33.084242+0000","last_became_active":"2024-03-21T08:43:33.078618+0000","last_became_peered":"2024-03-21T08:43:33.078618+0000","last_unstale":"2024-03-21T08:43:33.084242+0000","last_undegraded":"2024-03-21T08:43:33.084242+0000","last_fullsized":"2024-03-21T08:43:33.084242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084441+0000","last_change":"2024-03-21T08:43:34.082546+0000","last_active":"2024-03-21T08:43:34.084441+0000","last_peered":"2024-03-21T08:43:34.084441+0000","last_clean":"2024-03-21T08:43:34.084441+0000","last_became_active":"2024-03-21T08:43:34.082250+0000","last_became_peered":"2024-03-21T08:43:34.082250+0000","last_unstale":"2024-03-21T08:43:34.084441+0000","last_undegraded":"2024-03-21T08:43:34.084441+0000","last_fullsized":"2024-03-21T08:43:34.084441+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:35.232127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081037+0000","last_change":"2024-03-21T08:43:33.075815+0000","last_active":"2024-03-21T08:43:33.081037+0000","last_peered":"2024-03-21T08:43:33.081037+0000","last_clean":"2024-03-21T08:43:33.081037+0000","last_became_active":"2024-03-21T08:43:33.075533+0000","last_became_peered":"2024-03-21T08:43:33.075533+0000","last_unstale":"2024-03-21T08:43:33.081037+0000","last_undegraded":"2024-03-21T08:43:33.081037+0000","last_fullsized":"2024-03-21T08:43:33.081037+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063254+0000","last_change":"2024-03-21T08:43:33.063254+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063254+0000","last_undegraded":"2024-03-21T08:43:33.063254+0000","last_fullsized":"2024-03-21T08:43:33.063254+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081384+0000","last_change":"2024-03-21T08:43:33.076297+0000","last_active":"2024-03-21T08:43:33.081384+0000","last_peered":"2024-03-21T08:43:33.081384+0000","last_clean":"2024-03-21T08:43:33.081384+0000","last_became_active":"2024-03-21T08:43:33.075418+0000","last_became_peered":"2024-03-21T08:43:33.075418+0000","last_unstale":"2024-03-21T08:43:33.081384+0000","last_undegraded":"2024-03-21T08:43:33.081384+0000","last_fullsized":"2024-03-21T08:43:33.081384+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:12:14.723224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063758+0000","last_change":"2024-03-21T08:43:33.063758+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063758+0000","last_undegraded":"2024-03-21T08:43:33.063758+0000","last_fullsized":"2024-03-21T08:43:33.063758+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.066551+0000","last_change":"2024-03-21T08:43:33.084904+0000","last_active":"2024-03-21T08:43:34.066551+0000","last_peered":"2024-03-21T08:43:34.066551+0000","last_clean":"2024-03-21T08:43:34.066551+0000","last_became_active":"2024-03-21T08:43:33.084490+0000","last_became_peered":"2024-03-21T08:43:33.084490+0000","last_unstale":"2024-03-21T08:43:34.066551+0000","last_undegraded":"2024-03-21T08:43:34.066551+0000","last_fullsized":"2024-03-21T08:43:34.066551+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:59.934028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080558+0000","last_change":"2024-03-21T08:43:34.073629+0000","last_active":"2024-03-21T08:43:34.080558+0000","last_peered":"2024-03-21T08:43:34.080558+0000","last_clean":"2024-03-21T08:43:34.080558+0000","last_became_active":"2024-03-21T08:43:34.073264+0000","last_became_peered":"2024-03-21T08:43:34.073264+0000","last_unstale":"2024-03-21T08:43:34.080558+0000","last_undegraded":"2024-03-21T08:43:34.080558+0000","last_fullsized":"2024-03-21T08:43:34.080558+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:03:33.956736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084141+0000","last_change":"2024-03-21T08:43:33.080472+0000","last_active":"2024-03-21T08:43:33.084141+0000","last_peered":"2024-03-21T08:43:33.084141+0000","last_clean":"2024-03-21T08:43:33.084141+0000","last_became_active":"2024-03-21T08:43:33.080295+0000","last_became_peered":"2024-03-21T08:43:33.080295+0000","last_unstale":"2024-03-21T08:43:33.084141+0000","last_undegraded":"2024-03-21T08:43:33.084141+0000","last_fullsized":"2024-03-21T08:43:33.084141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084312+0000","last_change":"2024-03-21T08:43:34.082659+0000","last_active":"2024-03-21T08:43:34.084312+0000","last_peered":"2024-03-21T08:43:34.084312+0000","last_clean":"2024-03-21T08:43:34.084312+0000","last_became_active":"2024-03-21T08:43:34.082297+0000","last_became_peered":"2024-03-21T08:43:34.082297+0000","last_unstale":"2024-03-21T08:43:34.084312+0000","last_undegraded":"2024-03-21T08:43:34.084312+0000","last_fullsized":"2024-03-21T08:43:34.084312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:58:25.147246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"53'3","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100215+0000","last_change":"2024-03-21T08:43:33.079062+0000","last_active":"2024-03-21T08:43:34.100215+0000","last_peered":"2024-03-21T08:43:34.100215+0000","last_clean":"2024-03-21T08:43:34.100215+0000","last_became_active":"2024-03-21T08:43:33.078759+0000","last_became_peered":"2024-03-21T08:43:33.078759+0000","last_unstale":"2024-03-21T08:43:34.100215+0000","last_undegraded":"2024-03-21T08:43:34.100215+0000","last_fullsized":"2024-03-21T08:43:34.100215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:09.919722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079916+0000","last_change":"2024-03-21T08:43:34.074492+0000","last_active":"2024-03-21T08:43:34.079916+0000","last_peered":"2024-03-21T08:43:34.079916+0000","last_clean":"2024-03-21T08:43:34.079916+0000","last_became_active":"2024-03-21T08:43:34.074094+0000","last_became_peered":"2024-03-21T08:43:34.074094+0000","last_unstale":"2024-03-21T08:43:34.079916+0000","last_undegraded":"2024-03-21T08:43:34.079916+0000","last_fullsized":"2024-03-21T08:43:34.079916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:53:30.468809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081345+0000","last_change":"2024-03-21T08:43:33.074842+0000","last_active":"2024-03-21T08:43:33.081345+0000","last_peered":"2024-03-21T08:43:33.081345+0000","last_clean":"2024-03-21T08:43:33.081345+0000","last_became_active":"2024-03-21T08:43:33.074382+0000","last_became_peered":"2024-03-21T08:43:33.074382+0000","last_unstale":"2024-03-21T08:43:33.081345+0000","last_undegraded":"2024-03-21T08:43:33.081345+0000","last_fullsized":"2024-03-21T08:43:33.081345+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:20:56.748372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068759+0000","last_change":"2024-03-21T08:43:33.068759+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068759+0000","last_undegraded":"2024-03-21T08:43:33.068759+0000","last_fullsized":"2024-03-21T08:43:33.068759+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084080+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084080+0000","last_peered":"2024-03-21T08:43:33.084080+0000","last_clean":"2024-03-21T08:43:33.084080+0000","last_became_active":"2024-03-21T08:43:33.080224+0000","last_became_peered":"2024-03-21T08:43:33.080224+0000","last_unstale":"2024-03-21T08:43:33.084080+0000","last_undegraded":"2024-03-21T08:43:33.084080+0000","last_fullsized":"2024-03-21T08:43:33.084080+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080983+0000","last_change":"2024-03-21T08:43:34.078879+0000","last_active":"2024-03-21T08:43:34.080983+0000","last_peered":"2024-03-21T08:43:34.080983+0000","last_clean":"2024-03-21T08:43:34.080983+0000","last_became_active":"2024-03-21T08:43:34.078610+0000","last_became_peered":"2024-03-21T08:43:34.078610+0000","last_unstale":"2024-03-21T08:43:34.080983+0000","last_undegraded":"2024-03-21T08:43:34.080983+0000","last_fullsized":"2024-03-21T08:43:34.080983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:32.786072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072634+0000","last_change":"2024-03-21T08:43:33.085060+0000","last_active":"2024-03-21T08:43:34.072634+0000","last_peered":"2024-03-21T08:43:34.072634+0000","last_clean":"2024-03-21T08:43:34.072634+0000","last_became_active":"2024-03-21T08:43:33.084725+0000","last_became_peered":"2024-03-21T08:43:33.084725+0000","last_unstale":"2024-03-21T08:43:34.072634+0000","last_undegraded":"2024-03-21T08:43:34.072634+0000","last_fullsized":"2024-03-21T08:43:34.072634+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063545+0000","last_change":"2024-03-21T08:43:33.063545+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063545+0000","last_undegraded":"2024-03-21T08:43:33.063545+0000","last_fullsized":"2024-03-21T08:43:33.063545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081061+0000","last_change":"2024-03-21T08:43:33.075978+0000","last_active":"2024-03-21T08:43:33.081061+0000","last_peered":"2024-03-21T08:43:33.081061+0000","last_clean":"2024-03-21T08:43:33.081061+0000","last_became_active":"2024-03-21T08:43:33.075230+0000","last_became_peered":"2024-03-21T08:43:33.075230+0000","last_unstale":"2024-03-21T08:43:33.081061+0000","last_undegraded":"2024-03-21T08:43:33.081061+0000","last_fullsized":"2024-03-21T08:43:33.081061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.088395+0000","last_change":"2024-03-21T08:43:34.081971+0000","last_active":"2024-03-21T08:43:34.088395+0000","last_peered":"2024-03-21T08:43:34.088395+0000","last_clean":"2024-03-21T08:43:34.088395+0000","last_became_active":"2024-03-21T08:43:34.080960+0000","last_became_peered":"2024-03-21T08:43:34.080960+0000","last_unstale":"2024-03-21T08:43:34.088395+0000","last_undegraded":"2024-03-21T08:43:34.088395+0000","last_fullsized":"2024-03-21T08:43:34.088395+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:54.133660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"53'8","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.098747+0000","last_change":"2024-03-21T08:43:33.082199+0000","last_active":"2024-03-21T08:43:34.098747+0000","last_peered":"2024-03-21T08:43:34.098747+0000","last_clean":"2024-03-21T08:43:34.098747+0000","last_became_active":"2024-03-21T08:43:33.081892+0000","last_became_peered":"2024-03-21T08:43:33.081892+0000","last_unstale":"2024-03-21T08:43:34.098747+0000","last_undegraded":"2024-03-21T08:43:34.098747+0000","last_fullsized":"2024-03-21T08:43:34.098747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:19.722045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.092883+0000","last_change":"2024-03-21T08:43:34.086279+0000","last_active":"2024-03-21T08:43:34.092883+0000","last_peered":"2024-03-21T08:43:34.092883+0000","last_clean":"2024-03-21T08:43:34.092883+0000","last_became_active":"2024-03-21T08:43:34.086021+0000","last_became_peered":"2024-03-21T08:43:34.086021+0000","last_unstale":"2024-03-21T08:43:34.092883+0000","last_undegraded":"2024-03-21T08:43:34.092883+0000","last_fullsized":"2024-03-21T08:43:34.092883+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:51.667945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084347+0000","last_change":"2024-03-21T08:43:33.082173+0000","last_active":"2024-03-21T08:43:33.084347+0000","last_peered":"2024-03-21T08:43:33.084347+0000","last_clean":"2024-03-21T08:43:33.084347+0000","last_became_active":"2024-03-21T08:43:33.081902+0000","last_became_peered":"2024-03-21T08:43:33.081902+0000","last_unstale":"2024-03-21T08:43:33.084347+0000","last_undegraded":"2024-03-21T08:43:33.084347+0000","last_fullsized":"2024-03-21T08:43:33.084347+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065237+0000","last_change":"2024-03-21T08:43:33.065237+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065237+0000","last_undegraded":"2024-03-21T08:43:33.065237+0000","last_fullsized":"2024-03-21T08:43:33.065237+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083890+0000","last_change":"2024-03-21T08:43:33.077524+0000","last_active":"2024-03-21T08:43:33.083890+0000","last_peered":"2024-03-21T08:43:33.083890+0000","last_clean":"2024-03-21T08:43:33.083890+0000","last_became_active":"2024-03-21T08:43:33.077163+0000","last_became_peered":"2024-03-21T08:43:33.077163+0000","last_unstale":"2024-03-21T08:43:33.083890+0000","last_undegraded":"2024-03-21T08:43:33.083890+0000","last_fullsized":"2024-03-21T08:43:33.083890+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069726+0000","last_change":"2024-03-21T08:43:33.069726+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069726+0000","last_undegraded":"2024-03-21T08:43:33.069726+0000","last_fullsized":"2024-03-21T08:43:33.069726+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083836+0000","last_change":"2024-03-21T08:43:33.078372+0000","last_active":"2024-03-21T08:43:33.083836+0000","last_peered":"2024-03-21T08:43:33.083836+0000","last_clean":"2024-03-21T08:43:33.083836+0000","last_became_active":"2024-03-21T08:43:33.077797+0000","last_became_peered":"2024-03-21T08:43:33.077797+0000","last_unstale":"2024-03-21T08:43:33.083836+0000","last_undegraded":"2024-03-21T08:43:33.083836+0000","last_fullsized":"2024-03-21T08:43:33.083836+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.076176+0000","last_change":"2024-03-21T08:43:34.072036+0000","last_active":"2024-03-21T08:43:34.076176+0000","last_peered":"2024-03-21T08:43:34.076176+0000","last_clean":"2024-03-21T08:43:34.076176+0000","last_became_active":"2024-03-21T08:43:34.071710+0000","last_became_peered":"2024-03-21T08:43:34.071710+0000","last_unstale":"2024-03-21T08:43:34.076176+0000","last_undegraded":"2024-03-21T08:43:34.076176+0000","last_fullsized":"2024-03-21T08:43:34.076176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:37.084279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084283+0000","last_change":"2024-03-21T08:43:33.079120+0000","last_active":"2024-03-21T08:43:33.084283+0000","last_peered":"2024-03-21T08:43:33.084283+0000","last_clean":"2024-03-21T08:43:33.084283+0000","last_became_active":"2024-03-21T08:43:33.078672+0000","last_became_peered":"2024-03-21T08:43:33.078672+0000","last_unstale":"2024-03-21T08:43:33.084283+0000","last_undegraded":"2024-03-21T08:43:33.084283+0000","last_fullsized":"2024-03-21T08:43:33.084283+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068324+0000","last_change":"2024-03-21T08:43:33.068324+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068324+0000","last_undegraded":"2024-03-21T08:43:33.068324+0000","last_fullsized":"2024-03-21T08:43:33.068324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076791+0000","last_change":"2024-03-21T08:43:33.074311+0000","last_active":"2024-03-21T08:43:33.076791+0000","last_peered":"2024-03-21T08:43:33.076791+0000","last_clean":"2024-03-21T08:43:33.076791+0000","last_became_active":"2024-03-21T08:43:33.074047+0000","last_became_peered":"2024-03-21T08:43:33.074047+0000","last_unstale":"2024-03-21T08:43:33.076791+0000","last_undegraded":"2024-03-21T08:43:33.076791+0000","last_fullsized":"2024-03-21T08:43:33.076791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062142+0000","last_change":"2024-03-21T08:43:33.062142+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062142+0000","last_undegraded":"2024-03-21T08:43:33.062142+0000","last_fullsized":"2024-03-21T08:43:33.062142+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.062005+0000","last_change":"2024-03-21T08:43:33.080008+0000","last_active":"2024-03-21T08:43:34.062005+0000","last_peered":"2024-03-21T08:43:34.062005+0000","last_clean":"2024-03-21T08:43:34.062005+0000","last_became_active":"2024-03-21T08:43:33.079744+0000","last_became_peered":"2024-03-21T08:43:33.079744+0000","last_unstale":"2024-03-21T08:43:34.062005+0000","last_undegraded":"2024-03-21T08:43:34.062005+0000","last_fullsized":"2024-03-21T08:43:34.062005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:34.225900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062049+0000","last_change":"2024-03-21T08:43:33.062049+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062049+0000","last_undegraded":"2024-03-21T08:43:33.062049+0000","last_fullsized":"2024-03-21T08:43:33.062049+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081118+0000","last_change":"2024-03-21T08:43:33.076251+0000","last_active":"2024-03-21T08:43:33.081118+0000","last_peered":"2024-03-21T08:43:33.081118+0000","last_clean":"2024-03-21T08:43:33.081118+0000","last_became_active":"2024-03-21T08:43:33.074661+0000","last_became_peered":"2024-03-21T08:43:33.074661+0000","last_unstale":"2024-03-21T08:43:33.081118+0000","last_undegraded":"2024-03-21T08:43:33.081118+0000","last_fullsized":"2024-03-21T08:43:33.081118+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064097+0000","last_change":"2024-03-21T08:43:33.064097+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064097+0000","last_undegraded":"2024-03-21T08:43:33.064097+0000","last_fullsized":"2024-03-21T08:43:33.064097+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"53'2","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.106185+0000","last_change":"2024-03-21T08:43:33.079309+0000","last_active":"2024-03-21T08:43:34.106185+0000","last_peered":"2024-03-21T08:43:34.106185+0000","last_clean":"2024-03-21T08:43:34.106185+0000","last_became_active":"2024-03-21T08:43:33.079038+0000","last_became_peered":"2024-03-21T08:43:33.079038+0000","last_unstale":"2024-03-21T08:43:34.106185+0000","last_undegraded":"2024-03-21T08:43:34.106185+0000","last_fullsized":"2024-03-21T08:43:34.106185+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:23.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064702+0000","last_change":"2024-03-21T08:43:33.064702+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064702+0000","last_undegraded":"2024-03-21T08:43:33.064702+0000","last_fullsized":"2024-03-21T08:43:33.064702+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084195+0000","last_change":"2024-03-21T08:43:33.078636+0000","last_active":"2024-03-21T08:43:33.084195+0000","last_peered":"2024-03-21T08:43:33.084195+0000","last_clean":"2024-03-21T08:43:33.084195+0000","last_became_active":"2024-03-21T08:43:33.078336+0000","last_became_peered":"2024-03-21T08:43:33.078336+0000","last_unstale":"2024-03-21T08:43:33.084195+0000","last_undegraded":"2024-03-21T08:43:33.084195+0000","last_fullsized":"2024-03-21T08:43:33.084195+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084289+0000","last_change":"2024-03-21T08:43:34.082469+0000","last_active":"2024-03-21T08:43:34.084289+0000","last_peered":"2024-03-21T08:43:34.084289+0000","last_clean":"2024-03-21T08:43:34.084289+0000","last_became_active":"2024-03-21T08:43:34.082227+0000","last_became_peered":"2024-03-21T08:43:34.082227+0000","last_unstale":"2024-03-21T08:43:34.084289+0000","last_undegraded":"2024-03-21T08:43:34.084289+0000","last_fullsized":"2024-03-21T08:43:34.084289+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:14.199100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.071019+0000","last_change":"2024-03-21T08:43:33.077725+0000","last_active":"2024-03-21T08:43:34.071019+0000","last_peered":"2024-03-21T08:43:34.071019+0000","last_clean":"2024-03-21T08:43:34.071019+0000","last_became_active":"2024-03-21T08:43:33.077452+0000","last_became_peered":"2024-03-21T08:43:33.077452+0000","last_unstale":"2024-03-21T08:43:34.071019+0000","last_undegraded":"2024-03-21T08:43:34.071019+0000","last_fullsized":"2024-03-21T08:43:34.071019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:22:33.956371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064263+0000","last_change":"2024-03-21T08:43:33.064263+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064263+0000","last_undegraded":"2024-03-21T08:43:33.064263+0000","last_fullsized":"2024-03-21T08:43:33.064263+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081515+0000","last_change":"2024-03-21T08:43:33.076652+0000","last_active":"2024-03-21T08:43:33.081515+0000","last_peered":"2024-03-21T08:43:33.081515+0000","last_clean":"2024-03-21T08:43:33.081515+0000","last_became_active":"2024-03-21T08:43:33.074101+0000","last_became_peered":"2024-03-21T08:43:33.074101+0000","last_unstale":"2024-03-21T08:43:33.081515+0000","last_undegraded":"2024-03-21T08:43:33.081515+0000","last_fullsized":"2024-03-21T08:43:33.081515+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063671+0000","last_change":"2024-03-21T08:43:33.063671+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063671+0000","last_undegraded":"2024-03-21T08:43:33.063671+0000","last_fullsized":"2024-03-21T08:43:33.063671+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061812+0000","last_change":"2024-03-21T08:43:33.081506+0000","last_active":"2024-03-21T08:43:34.061812+0000","last_peered":"2024-03-21T08:43:34.061812+0000","last_clean":"2024-03-21T08:43:34.061812+0000","last_became_active":"2024-03-21T08:43:33.080894+0000","last_became_peered":"2024-03-21T08:43:33.080894+0000","last_unstale":"2024-03-21T08:43:34.061812+0000","last_undegraded":"2024-03-21T08:43:34.061812+0000","last_fullsized":"2024-03-21T08:43:34.061812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093878+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:34.093878+0000","last_peered":"2024-03-21T08:43:34.093878+0000","last_clean":"2024-03-21T08:43:34.093878+0000","last_became_active":"2024-03-21T08:43:34.089436+0000","last_became_peered":"2024-03-21T08:43:34.089436+0000","last_unstale":"2024-03-21T08:43:34.093878+0000","last_undegraded":"2024-03-21T08:43:34.093878+0000","last_fullsized":"2024-03-21T08:43:34.093878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:10.546830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084246+0000","last_change":"2024-03-21T08:43:33.080132+0000","last_active":"2024-03-21T08:43:33.084246+0000","last_peered":"2024-03-21T08:43:33.084246+0000","last_clean":"2024-03-21T08:43:33.084246+0000","last_became_active":"2024-03-21T08:43:33.079867+0000","last_became_peered":"2024-03-21T08:43:33.079867+0000","last_unstale":"2024-03-21T08:43:33.084246+0000","last_undegraded":"2024-03-21T08:43:33.084246+0000","last_fullsized":"2024-03-21T08:43:33.084246+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:33:42.756681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.089682+0000","last_change":"2024-03-21T08:43:34.084715+0000","last_active":"2024-03-21T08:43:34.089682+0000","last_peered":"2024-03-21T08:43:34.089682+0000","last_clean":"2024-03-21T08:43:34.089682+0000","last_became_active":"2024-03-21T08:43:34.084436+0000","last_became_peered":"2024-03-21T08:43:34.084436+0000","last_unstale":"2024-03-21T08:43:34.089682+0000","last_undegraded":"2024-03-21T08:43:34.089682+0000","last_fullsized":"2024-03-21T08:43:34.089682+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:33:01.770928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061783+0000","last_change":"2024-03-21T08:43:33.076471+0000","last_active":"2024-03-21T08:43:34.061783+0000","last_peered":"2024-03-21T08:43:34.061783+0000","last_clean":"2024-03-21T08:43:34.061783+0000","last_became_active":"2024-03-21T08:43:33.076185+0000","last_became_peered":"2024-03-21T08:43:33.076185+0000","last_unstale":"2024-03-21T08:43:34.061783+0000","last_undegraded":"2024-03-21T08:43:34.061783+0000","last_fullsized":"2024-03-21T08:43:34.061783+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062860+0000","last_change":"2024-03-21T08:43:33.062860+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062860+0000","last_undegraded":"2024-03-21T08:43:33.062860+0000","last_fullsized":"2024-03-21T08:43:33.062860+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076315+0000","last_change":"2024-03-21T08:43:33.074819+0000","last_active":"2024-03-21T08:43:33.076315+0000","last_peered":"2024-03-21T08:43:33.076315+0000","last_clean":"2024-03-21T08:43:33.076315+0000","last_became_active":"2024-03-21T08:43:33.074585+0000","last_became_peered":"2024-03-21T08:43:33.074585+0000","last_unstale":"2024-03-21T08:43:33.076315+0000","last_undegraded":"2024-03-21T08:43:33.076315+0000","last_fullsized":"2024-03-21T08:43:33.076315+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063064+0000","last_change":"2024-03-21T08:43:33.063064+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063064+0000","last_undegraded":"2024-03-21T08:43:33.063064+0000","last_fullsized":"2024-03-21T08:43:33.063064+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072378+0000","last_change":"2024-03-21T08:43:33.081102+0000","last_active":"2024-03-21T08:43:34.072378+0000","last_peered":"2024-03-21T08:43:34.072378+0000","last_clean":"2024-03-21T08:43:34.072378+0000","last_became_active":"2024-03-21T08:43:33.080820+0000","last_became_peered":"2024-03-21T08:43:33.080820+0000","last_unstale":"2024-03-21T08:43:34.072378+0000","last_undegraded":"2024-03-21T08:43:34.072378+0000","last_fullsized":"2024-03-21T08:43:34.072378+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.065013+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:43:34.065013+0000","last_peered":"2024-03-21T08:43:34.065013+0000","last_clean":"2024-03-21T08:43:34.065013+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:43:34.065013+0000","last_undegraded":"2024-03-21T08:43:34.065013+0000","last_fullsized":"2024-03-21T08:43:34.065013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509476,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1228800,"data_stored":871737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":281209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":281299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":602112,"data_stored":267383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":864410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:43:38.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:43:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.762+0000 7fa257000640 1 Processor -- start 2024-03-21T08:43:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa257000640 1 -- start start 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa257000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 0x7fa2500caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa257000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500cd850 con 0x7fa25015de50 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa257000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500cdb30 con 0x7fa2500cd0d0 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa257000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25015e230 con 0x7fa2500af910 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa25015de50 0x7fa250162210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38696/0 (socket says 172.21.15.28:38696) 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24fe00640 1 -- 172.21.15.28:0/4204600937 learned_addr learned my addr 172.21.15.28:0/4204600937 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa255000640 1 --2- 172.21.15.28:0/4204600937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 0x7fa2500caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743363852 0 0) 0x7fa2500cd850 con 0x7fa25015de50 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa234003530 con 0x7fa25015de50 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646189534 0 0) 0x7fa2500cdb30 con 0x7fa2500cd0d0 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2500cd850 con 0x7fa2500cd0d0 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa255000640 1 -- 172.21.15.28:0/4204600937 >> v1:172.21.15.151:6789/0 conn(0x7fa2500cd0d0 legacy=0x7fa2500cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa255000640 1 -- 172.21.15.28:0/4204600937 >> v1:172.21.15.28:6789/0 conn(0x7fa25015de50 legacy=0x7fa250162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.766+0000 7fa255000640 1 -- 172.21.15.28:0/4204600937 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250161a10 con 0x7fa2500af910 2024-03-21T08:43:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa255000640 1 --2- 172.21.15.28:0/4204600937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 0x7fa2500caf40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa248009dc0 tx=0x7fa24801c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf9ae9f308d241 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b2070 con 0x7fa2500af910 2024-03-21T08:43:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2480a57c0 con 0x7fa2500af910 2024-03-21T08:43:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa24f400640 1 -- 172.21.15.28:0/4204600937 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480a5b90 con 0x7fa2500af910 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 -- 172.21.15.28:0/4204600937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 msgr2=0x7fa2500caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 --2- 172.21.15.28:0/4204600937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 0x7fa2500caf40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa248009dc0 tx=0x7fa24801c900 comp rx=0 tx=0).stop 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 -- 172.21.15.28:0/4204600937 shutdown_connections 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 --2- 172.21.15.28:0/4204600937 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2500af910 0x7fa2500caf40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 -- 172.21.15.28:0/4204600937 >> 172.21.15.28:0/4204600937 conn(0x7fa250099a30 msgr2=0x7fa250099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 -- 172.21.15.28:0/4204600937 shutdown_connections 2024-03-21T08:43:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.770+0000 7fa257000640 1 -- 172.21.15.28:0/4204600937 wait complete. 2024-03-21T08:43:38.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 Processor -- start 2024-03-21T08:43:38.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- start start 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 0x7fa2500b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa25015de50 0x7fa250159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25016c5c0 con 0x7fa2500af910 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25016c440 con 0x7fa2500cd0d0 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25016c740 con 0x7fa25015de50 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43484/0 (socket says 172.21.15.28:43484) 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 -- 172.21.15.28:0/2929103837 learned_addr learned my addr 172.21.15.28:0/2929103837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa247400640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 0x7fa2500b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa255000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa25015de50 0x7fa250159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 -- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa25015de50 msgr2=0x7fa250159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa25015de50 0x7fa250159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 -- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 msgr2=0x7fa2500b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 0x7fa2500b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 -- 172.21.15.28:0/2929103837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500c57f0 con 0x7fa2500af910 2024-03-21T08:43:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa247400640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 0x7fa2500b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24fe00640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c420 tx=0x7fa24000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240017070 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240005cf0 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240007310 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2500c5ad0 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2500c6090 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.774+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25004f410 con 0x7fa2500af910 2024-03-21T08:43:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.778+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa24001a070 con 0x7fa2500af910 2024-03-21T08:43:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.778+0000 7fa24cc00640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 0x7fa22c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.778+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa2400854f0 con 0x7fa2500af910 2024-03-21T08:43:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.778+0000 7fa247400640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 0x7fa22c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.778+0000 7fa247400640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 0x7fa22c069fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa25006f770 tx=0x7fa238009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.782+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa240055d60 con 0x7fa2500af910 2024-03-21T08:43:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.942+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2500cbf70 con 0x7fa22c067b20 2024-03-21T08:43:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.950+0000 7fa24cc00640 1 -- 172.21.15.28:0/2929103837 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+248843 (secure 0 0 0) 0x7fa2500cbf70 con 0x7fa22c067b20 2024-03-21T08:43:38.956 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:38.957 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 msgr2=0x7fa22c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 0x7fa22c069fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa25006f770 tx=0x7fa238009040 comp rx=0 tx=0).stop 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 msgr2=0x7fa25015f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa24000c420 tx=0x7fa24000c8e0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 shutdown_connections 2024-03-21T08:43:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa22c067b20 0x7fa22c069fe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa25015de50 0x7fa250159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2500cd0d0 0x7fa2500b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 --2- 172.21.15.28:0/2929103837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2500af910 0x7fa25015f490 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 >> 172.21.15.28:0/2929103837 conn(0x7fa250099a30 msgr2=0x7fa2500b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 shutdown_connections 2024-03-21T08:43:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:38.954+0000 7fa257000640 1 -- 172.21.15.28:0/2929103837 wait complete. 2024-03-21T08:43:38.976 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-21T08:43:37.753502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":178,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238812,"kb_used_data":5996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735908,"statfs":{"total":773094113280,"available":772849569792,"internally_reserved":0,"allocated":6139904,"data_stored":3388164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705743"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.105337+0000","last_change":"2024-03-21T08:43:34.104498+0000","last_active":"2024-03-21T08:43:34.105337+0000","last_peered":"2024-03-21T08:43:34.105337+0000","last_clean":"2024-03-21T08:43:34.105337+0000","last_became_active":"2024-03-21T08:43:34.102336+0000","last_became_peered":"2024-03-21T08:43:34.102336+0000","last_unstale":"2024-03-21T08:43:34.105337+0000","last_undegraded":"2024-03-21T08:43:34.105337+0000","last_fullsized":"2024-03-21T08:43:34.105337+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:44:57.769559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.066283+0000","last_change":"2024-03-21T08:43:33.066283+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.066283+0000","last_undegraded":"2024-03-21T08:43:33.066283+0000","last_fullsized":"2024-03-21T08:43:33.066283+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084512+0000","last_change":"2024-03-21T08:43:34.079798+0000","last_active":"2024-03-21T08:43:34.084512+0000","last_peered":"2024-03-21T08:43:34.084512+0000","last_clean":"2024-03-21T08:43:34.084512+0000","last_became_active":"2024-03-21T08:43:34.079557+0000","last_became_peered":"2024-03-21T08:43:34.079557+0000","last_unstale":"2024-03-21T08:43:34.084512+0000","last_undegraded":"2024-03-21T08:43:34.084512+0000","last_fullsized":"2024-03-21T08:43:34.084512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:33:45.824524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065481+0000","last_change":"2024-03-21T08:43:33.065481+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065481+0000","last_undegraded":"2024-03-21T08:43:33.065481+0000","last_fullsized":"2024-03-21T08:43:33.065481+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074891+0000","last_change":"2024-03-21T08:43:34.069862+0000","last_active":"2024-03-21T08:43:34.074891+0000","last_peered":"2024-03-21T08:43:34.074891+0000","last_clean":"2024-03-21T08:43:34.074891+0000","last_became_active":"2024-03-21T08:43:34.069561+0000","last_became_peered":"2024-03-21T08:43:34.069561+0000","last_unstale":"2024-03-21T08:43:34.074891+0000","last_undegraded":"2024-03-21T08:43:34.074891+0000","last_fullsized":"2024-03-21T08:43:34.074891+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:02:24.282774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064693+0000","last_change":"2024-03-21T08:43:33.064693+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064693+0000","last_undegraded":"2024-03-21T08:43:33.064693+0000","last_fullsized":"2024-03-21T08:43:33.064693+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065768+0000","last_change":"2024-03-21T08:43:33.065768+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065768+0000","last_undegraded":"2024-03-21T08:43:33.065768+0000","last_fullsized":"2024-03-21T08:43:33.065768+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062573+0000","last_change":"2024-03-21T08:43:33.062573+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062573+0000","last_undegraded":"2024-03-21T08:43:33.062573+0000","last_fullsized":"2024-03-21T08:43:33.062573+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063862+0000","last_change":"2024-03-21T08:43:33.063862+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063862+0000","last_undegraded":"2024-03-21T08:43:33.063862+0000","last_fullsized":"2024-03-21T08:43:33.063862+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079493+0000","last_change":"2024-03-21T08:43:34.076812+0000","last_active":"2024-03-21T08:43:34.079493+0000","last_peered":"2024-03-21T08:43:34.079493+0000","last_clean":"2024-03-21T08:43:34.079493+0000","last_became_active":"2024-03-21T08:43:34.076556+0000","last_became_peered":"2024-03-21T08:43:34.076556+0000","last_unstale":"2024-03-21T08:43:34.079493+0000","last_undegraded":"2024-03-21T08:43:34.079493+0000","last_fullsized":"2024-03-21T08:43:34.079493+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:17:35.654015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.099722+0000","last_change":"2024-03-21T08:43:34.096375+0000","last_active":"2024-03-21T08:43:34.099722+0000","last_peered":"2024-03-21T08:43:34.099722+0000","last_clean":"2024-03-21T08:43:34.099722+0000","last_became_active":"2024-03-21T08:43:34.095943+0000","last_became_peered":"2024-03-21T08:43:34.095943+0000","last_unstale":"2024-03-21T08:43:34.099722+0000","last_undegraded":"2024-03-21T08:43:34.099722+0000","last_fullsized":"2024-03-21T08:43:34.099722+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:20.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065944+0000","last_change":"2024-03-21T08:43:33.065944+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065944+0000","last_undegraded":"2024-03-21T08:43:33.065944+0000","last_fullsized":"2024-03-21T08:43:33.065944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094467+0000","last_change":"2024-03-21T08:43:34.090738+0000","last_active":"2024-03-21T08:43:34.094467+0000","last_peered":"2024-03-21T08:43:34.094467+0000","last_clean":"2024-03-21T08:43:34.094467+0000","last_became_active":"2024-03-21T08:43:34.090493+0000","last_became_peered":"2024-03-21T08:43:34.090493+0000","last_unstale":"2024-03-21T08:43:34.094467+0000","last_undegraded":"2024-03-21T08:43:34.094467+0000","last_fullsized":"2024-03-21T08:43:34.094467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063655+0000","last_change":"2024-03-21T08:43:33.063655+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063655+0000","last_undegraded":"2024-03-21T08:43:33.063655+0000","last_fullsized":"2024-03-21T08:43:33.063655+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063371+0000","last_change":"2024-03-21T08:43:33.063371+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063371+0000","last_undegraded":"2024-03-21T08:43:33.063371+0000","last_fullsized":"2024-03-21T08:43:33.063371+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065030+0000","last_change":"2024-03-21T08:43:33.065030+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065030+0000","last_undegraded":"2024-03-21T08:43:33.065030+0000","last_fullsized":"2024-03-21T08:43:33.065030+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068546+0000","last_change":"2024-03-21T08:43:33.068546+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068546+0000","last_undegraded":"2024-03-21T08:43:33.068546+0000","last_fullsized":"2024-03-21T08:43:33.068546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093933+0000","last_change":"2024-03-21T08:43:34.090757+0000","last_active":"2024-03-21T08:43:34.093933+0000","last_peered":"2024-03-21T08:43:34.093933+0000","last_clean":"2024-03-21T08:43:34.093933+0000","last_became_active":"2024-03-21T08:43:34.090125+0000","last_became_peered":"2024-03-21T08:43:34.090125+0000","last_unstale":"2024-03-21T08:43:34.093933+0000","last_undegraded":"2024-03-21T08:43:34.093933+0000","last_fullsized":"2024-03-21T08:43:34.093933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:14:11.321572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.103454+0000","last_change":"2024-03-21T08:43:34.099533+0000","last_active":"2024-03-21T08:43:34.103454+0000","last_peered":"2024-03-21T08:43:34.103454+0000","last_clean":"2024-03-21T08:43:34.103454+0000","last_became_active":"2024-03-21T08:43:34.099273+0000","last_became_peered":"2024-03-21T08:43:34.099273+0000","last_unstale":"2024-03-21T08:43:34.103454+0000","last_undegraded":"2024-03-21T08:43:34.103454+0000","last_fullsized":"2024-03-21T08:43:34.103454+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:25:59.522961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.070108+0000","last_change":"2024-03-21T08:43:33.070108+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.070108+0000","last_undegraded":"2024-03-21T08:43:33.070108+0000","last_fullsized":"2024-03-21T08:43:33.070108+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073021+0000","last_change":"2024-03-21T08:43:34.069515+0000","last_active":"2024-03-21T08:43:34.073021+0000","last_peered":"2024-03-21T08:43:34.073021+0000","last_clean":"2024-03-21T08:43:34.073021+0000","last_became_active":"2024-03-21T08:43:34.069239+0000","last_became_peered":"2024-03-21T08:43:34.069239+0000","last_unstale":"2024-03-21T08:43:34.073021+0000","last_undegraded":"2024-03-21T08:43:34.073021+0000","last_fullsized":"2024-03-21T08:43:34.073021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:40:59.946986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064406+0000","last_change":"2024-03-21T08:43:33.064406+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064406+0000","last_undegraded":"2024-03-21T08:43:33.064406+0000","last_fullsized":"2024-03-21T08:43:33.064406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100244+0000","last_change":"2024-03-21T08:43:34.097093+0000","last_active":"2024-03-21T08:43:34.100244+0000","last_peered":"2024-03-21T08:43:34.100244+0000","last_clean":"2024-03-21T08:43:34.100244+0000","last_became_active":"2024-03-21T08:43:34.096448+0000","last_became_peered":"2024-03-21T08:43:34.096448+0000","last_unstale":"2024-03-21T08:43:34.100244+0000","last_undegraded":"2024-03-21T08:43:34.100244+0000","last_fullsized":"2024-03-21T08:43:34.100244+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:20:52.859039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094188+0000","last_change":"2024-03-21T08:43:34.089864+0000","last_active":"2024-03-21T08:43:34.094188+0000","last_peered":"2024-03-21T08:43:34.094188+0000","last_clean":"2024-03-21T08:43:34.094188+0000","last_became_active":"2024-03-21T08:43:34.089605+0000","last_became_peered":"2024-03-21T08:43:34.089605+0000","last_unstale":"2024-03-21T08:43:34.094188+0000","last_undegraded":"2024-03-21T08:43:34.094188+0000","last_fullsized":"2024-03-21T08:43:34.094188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:24.367969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074927+0000","last_change":"2024-03-21T08:43:34.070708+0000","last_active":"2024-03-21T08:43:34.074927+0000","last_peered":"2024-03-21T08:43:34.074927+0000","last_clean":"2024-03-21T08:43:34.074927+0000","last_became_active":"2024-03-21T08:43:34.070458+0000","last_became_peered":"2024-03-21T08:43:34.070458+0000","last_unstale":"2024-03-21T08:43:34.074927+0000","last_undegraded":"2024-03-21T08:43:34.074927+0000","last_fullsized":"2024-03-21T08:43:34.074927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:48.402969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069742+0000","last_change":"2024-03-21T08:43:33.069742+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069742+0000","last_undegraded":"2024-03-21T08:43:33.069742+0000","last_fullsized":"2024-03-21T08:43:33.069742+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065512+0000","last_change":"2024-03-21T08:43:33.065512+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065512+0000","last_undegraded":"2024-03-21T08:43:33.065512+0000","last_fullsized":"2024-03-21T08:43:33.065512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080452+0000","last_change":"2024-03-21T08:43:34.074171+0000","last_active":"2024-03-21T08:43:34.080452+0000","last_peered":"2024-03-21T08:43:34.080452+0000","last_clean":"2024-03-21T08:43:34.080452+0000","last_became_active":"2024-03-21T08:43:34.073918+0000","last_became_peered":"2024-03-21T08:43:34.073918+0000","last_unstale":"2024-03-21T08:43:34.080452+0000","last_undegraded":"2024-03-21T08:43:34.080452+0000","last_fullsized":"2024-03-21T08:43:34.080452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:23:54.394841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068308+0000","last_change":"2024-03-21T08:43:33.068308+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068308+0000","last_undegraded":"2024-03-21T08:43:33.068308+0000","last_fullsized":"2024-03-21T08:43:33.068308+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.085472+0000","last_change":"2024-03-21T08:43:34.079738+0000","last_active":"2024-03-21T08:43:34.085472+0000","last_peered":"2024-03-21T08:43:34.085472+0000","last_clean":"2024-03-21T08:43:34.085472+0000","last_became_active":"2024-03-21T08:43:34.079479+0000","last_became_peered":"2024-03-21T08:43:34.079479+0000","last_unstale":"2024-03-21T08:43:34.085472+0000","last_undegraded":"2024-03-21T08:43:34.085472+0000","last_fullsized":"2024-03-21T08:43:34.085472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:22:25.192237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073626+0000","last_change":"2024-03-21T08:43:34.069752+0000","last_active":"2024-03-21T08:43:34.073626+0000","last_peered":"2024-03-21T08:43:34.073626+0000","last_clean":"2024-03-21T08:43:34.073626+0000","last_became_active":"2024-03-21T08:43:34.068765+0000","last_became_peered":"2024-03-21T08:43:34.068765+0000","last_unstale":"2024-03-21T08:43:34.073626+0000","last_undegraded":"2024-03-21T08:43:34.073626+0000","last_fullsized":"2024-03-21T08:43:34.073626+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:28.749067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063487+0000","last_change":"2024-03-21T08:43:33.063487+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063487+0000","last_undegraded":"2024-03-21T08:43:33.063487+0000","last_fullsized":"2024-03-21T08:43:33.063487+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064212+0000","last_change":"2024-03-21T08:43:33.064212+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064212+0000","last_undegraded":"2024-03-21T08:43:33.064212+0000","last_fullsized":"2024-03-21T08:43:33.064212+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084794+0000","last_change":"2024-03-21T08:43:33.081655+0000","last_active":"2024-03-21T08:43:33.084794+0000","last_peered":"2024-03-21T08:43:33.084794+0000","last_clean":"2024-03-21T08:43:33.084794+0000","last_became_active":"2024-03-21T08:43:33.081260+0000","last_became_peered":"2024-03-21T08:43:33.081260+0000","last_unstale":"2024-03-21T08:43:33.084794+0000","last_undegraded":"2024-03-21T08:43:33.084794+0000","last_fullsized":"2024-03-21T08:43:33.084794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074406+0000","last_change":"2024-03-21T08:43:34.070772+0000","last_active":"2024-03-21T08:43:34.074406+0000","last_peered":"2024-03-21T08:43:34.074406+0000","last_clean":"2024-03-21T08:43:34.074406+0000","last_became_active":"2024-03-21T08:43:34.070484+0000","last_became_peered":"2024-03-21T08:43:34.070484+0000","last_unstale":"2024-03-21T08:43:34.074406+0000","last_undegraded":"2024-03-21T08:43:34.074406+0000","last_fullsized":"2024-03-21T08:43:34.074406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084121+0000","last_change":"2024-03-21T08:43:33.078482+0000","last_active":"2024-03-21T08:43:33.084121+0000","last_peered":"2024-03-21T08:43:33.084121+0000","last_clean":"2024-03-21T08:43:33.084121+0000","last_became_active":"2024-03-21T08:43:33.078221+0000","last_became_peered":"2024-03-21T08:43:33.078221+0000","last_unstale":"2024-03-21T08:43:33.084121+0000","last_undegraded":"2024-03-21T08:43:33.084121+0000","last_fullsized":"2024-03-21T08:43:33.084121+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062843+0000","last_change":"2024-03-21T08:43:33.062843+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062843+0000","last_undegraded":"2024-03-21T08:43:33.062843+0000","last_fullsized":"2024-03-21T08:43:33.062843+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084733+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084733+0000","last_peered":"2024-03-21T08:43:33.084733+0000","last_clean":"2024-03-21T08:43:33.084733+0000","last_became_active":"2024-03-21T08:43:33.080225+0000","last_became_peered":"2024-03-21T08:43:33.080225+0000","last_unstale":"2024-03-21T08:43:33.084733+0000","last_undegraded":"2024-03-21T08:43:33.084733+0000","last_fullsized":"2024-03-21T08:43:33.084733+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065020+0000","last_change":"2024-03-21T08:43:33.065020+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065020+0000","last_undegraded":"2024-03-21T08:43:33.065020+0000","last_fullsized":"2024-03-21T08:43:33.065020+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.086635+0000","last_change":"2024-03-21T08:43:33.083210+0000","last_active":"2024-03-21T08:43:33.086635+0000","last_peered":"2024-03-21T08:43:33.086635+0000","last_clean":"2024-03-21T08:43:33.086635+0000","last_became_active":"2024-03-21T08:43:33.082822+0000","last_became_peered":"2024-03-21T08:43:33.082822+0000","last_unstale":"2024-03-21T08:43:33.086635+0000","last_undegraded":"2024-03-21T08:43:33.086635+0000","last_fullsized":"2024-03-21T08:43:33.086635+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:26:53.025631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064047+0000","last_change":"2024-03-21T08:43:33.064047+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064047+0000","last_undegraded":"2024-03-21T08:43:33.064047+0000","last_fullsized":"2024-03-21T08:43:33.064047+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084144+0000","last_change":"2024-03-21T08:43:33.078204+0000","last_active":"2024-03-21T08:43:33.084144+0000","last_peered":"2024-03-21T08:43:33.084144+0000","last_clean":"2024-03-21T08:43:33.084144+0000","last_became_active":"2024-03-21T08:43:33.077529+0000","last_became_peered":"2024-03-21T08:43:33.077529+0000","last_unstale":"2024-03-21T08:43:33.084144+0000","last_undegraded":"2024-03-21T08:43:33.084144+0000","last_fullsized":"2024-03-21T08:43:33.084144+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:35:40.609258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069118+0000","last_change":"2024-03-21T08:43:33.069118+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069118+0000","last_undegraded":"2024-03-21T08:43:33.069118+0000","last_fullsized":"2024-03-21T08:43:33.069118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084242+0000","last_change":"2024-03-21T08:43:33.078883+0000","last_active":"2024-03-21T08:43:33.084242+0000","last_peered":"2024-03-21T08:43:33.084242+0000","last_clean":"2024-03-21T08:43:33.084242+0000","last_became_active":"2024-03-21T08:43:33.078618+0000","last_became_peered":"2024-03-21T08:43:33.078618+0000","last_unstale":"2024-03-21T08:43:33.084242+0000","last_undegraded":"2024-03-21T08:43:33.084242+0000","last_fullsized":"2024-03-21T08:43:33.084242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084441+0000","last_change":"2024-03-21T08:43:34.082546+0000","last_active":"2024-03-21T08:43:34.084441+0000","last_peered":"2024-03-21T08:43:34.084441+0000","last_clean":"2024-03-21T08:43:34.084441+0000","last_became_active":"2024-03-21T08:43:34.082250+0000","last_became_peered":"2024-03-21T08:43:34.082250+0000","last_unstale":"2024-03-21T08:43:34.084441+0000","last_undegraded":"2024-03-21T08:43:34.084441+0000","last_fullsized":"2024-03-21T08:43:34.084441+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:35.232127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081037+0000","last_change":"2024-03-21T08:43:33.075815+0000","last_active":"2024-03-21T08:43:33.081037+0000","last_peered":"2024-03-21T08:43:33.081037+0000","last_clean":"2024-03-21T08:43:33.081037+0000","last_became_active":"2024-03-21T08:43:33.075533+0000","last_became_peered":"2024-03-21T08:43:33.075533+0000","last_unstale":"2024-03-21T08:43:33.081037+0000","last_undegraded":"2024-03-21T08:43:33.081037+0000","last_fullsized":"2024-03-21T08:43:33.081037+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063254+0000","last_change":"2024-03-21T08:43:33.063254+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063254+0000","last_undegraded":"2024-03-21T08:43:33.063254+0000","last_fullsized":"2024-03-21T08:43:33.063254+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081384+0000","last_change":"2024-03-21T08:43:33.076297+0000","last_active":"2024-03-21T08:43:33.081384+0000","last_peered":"2024-03-21T08:43:33.081384+0000","last_clean":"2024-03-21T08:43:33.081384+0000","last_became_active":"2024-03-21T08:43:33.075418+0000","last_became_peered":"2024-03-21T08:43:33.075418+0000","last_unstale":"2024-03-21T08:43:33.081384+0000","last_undegraded":"2024-03-21T08:43:33.081384+0000","last_fullsized":"2024-03-21T08:43:33.081384+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:12:14.723224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063758+0000","last_change":"2024-03-21T08:43:33.063758+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063758+0000","last_undegraded":"2024-03-21T08:43:33.063758+0000","last_fullsized":"2024-03-21T08:43:33.063758+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.066551+0000","last_change":"2024-03-21T08:43:33.084904+0000","last_active":"2024-03-21T08:43:34.066551+0000","last_peered":"2024-03-21T08:43:34.066551+0000","last_clean":"2024-03-21T08:43:34.066551+0000","last_became_active":"2024-03-21T08:43:33.084490+0000","last_became_peered":"2024-03-21T08:43:33.084490+0000","last_unstale":"2024-03-21T08:43:34.066551+0000","last_undegraded":"2024-03-21T08:43:34.066551+0000","last_fullsized":"2024-03-21T08:43:34.066551+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:59.934028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080558+0000","last_change":"2024-03-21T08:43:34.073629+0000","last_active":"2024-03-21T08:43:34.080558+0000","last_peered":"2024-03-21T08:43:34.080558+0000","last_clean":"2024-03-21T08:43:34.080558+0000","last_became_active":"2024-03-21T08:43:34.073264+0000","last_became_peered":"2024-03-21T08:43:34.073264+0000","last_unstale":"2024-03-21T08:43:34.080558+0000","last_undegraded":"2024-03-21T08:43:34.080558+0000","last_fullsized":"2024-03-21T08:43:34.080558+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:03:33.956736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084141+0000","last_change":"2024-03-21T08:43:33.080472+0000","last_active":"2024-03-21T08:43:33.084141+0000","last_peered":"2024-03-21T08:43:33.084141+0000","last_clean":"2024-03-21T08:43:33.084141+0000","last_became_active":"2024-03-21T08:43:33.080295+0000","last_became_peered":"2024-03-21T08:43:33.080295+0000","last_unstale":"2024-03-21T08:43:33.084141+0000","last_undegraded":"2024-03-21T08:43:33.084141+0000","last_fullsized":"2024-03-21T08:43:33.084141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084312+0000","last_change":"2024-03-21T08:43:34.082659+0000","last_active":"2024-03-21T08:43:34.084312+0000","last_peered":"2024-03-21T08:43:34.084312+0000","last_clean":"2024-03-21T08:43:34.084312+0000","last_became_active":"2024-03-21T08:43:34.082297+0000","last_became_peered":"2024-03-21T08:43:34.082297+0000","last_unstale":"2024-03-21T08:43:34.084312+0000","last_undegraded":"2024-03-21T08:43:34.084312+0000","last_fullsized":"2024-03-21T08:43:34.084312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:58:25.147246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"53'3","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100215+0000","last_change":"2024-03-21T08:43:33.079062+0000","last_active":"2024-03-21T08:43:34.100215+0000","last_peered":"2024-03-21T08:43:34.100215+0000","last_clean":"2024-03-21T08:43:34.100215+0000","last_became_active":"2024-03-21T08:43:33.078759+0000","last_became_peered":"2024-03-21T08:43:33.078759+0000","last_unstale":"2024-03-21T08:43:34.100215+0000","last_undegraded":"2024-03-21T08:43:34.100215+0000","last_fullsized":"2024-03-21T08:43:34.100215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:09.919722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079916+0000","last_change":"2024-03-21T08:43:34.074492+0000","last_active":"2024-03-21T08:43:34.079916+0000","last_peered":"2024-03-21T08:43:34.079916+0000","last_clean":"2024-03-21T08:43:34.079916+0000","last_became_active":"2024-03-21T08:43:34.074094+0000","last_became_peered":"2024-03-21T08:43:34.074094+0000","last_unstale":"2024-03-21T08:43:34.079916+0000","last_undegraded":"2024-03-21T08:43:34.079916+0000","last_fullsized":"2024-03-21T08:43:34.079916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:53:30.468809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081345+0000","last_change":"2024-03-21T08:43:33.074842+0000","last_active":"2024-03-21T08:43:33.081345+0000","last_peered":"2024-03-21T08:43:33.081345+0000","last_clean":"2024-03-21T08:43:33.081345+0000","last_became_active":"2024-03-21T08:43:33.074382+0000","last_became_peered":"2024-03-21T08:43:33.074382+0000","last_unstale":"2024-03-21T08:43:33.081345+0000","last_undegraded":"2024-03-21T08:43:33.081345+0000","last_fullsized":"2024-03-21T08:43:33.081345+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:20:56.748372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068759+0000","last_change":"2024-03-21T08:43:33.068759+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068759+0000","last_undegraded":"2024-03-21T08:43:33.068759+0000","last_fullsized":"2024-03-21T08:43:33.068759+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084080+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:33.084080+0000","last_peered":"2024-03-21T08:43:33.084080+0000","last_clean":"2024-03-21T08:43:33.084080+0000","last_became_active":"2024-03-21T08:43:33.080224+0000","last_became_peered":"2024-03-21T08:43:33.080224+0000","last_unstale":"2024-03-21T08:43:33.084080+0000","last_undegraded":"2024-03-21T08:43:33.084080+0000","last_fullsized":"2024-03-21T08:43:33.084080+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080983+0000","last_change":"2024-03-21T08:43:34.078879+0000","last_active":"2024-03-21T08:43:34.080983+0000","last_peered":"2024-03-21T08:43:34.080983+0000","last_clean":"2024-03-21T08:43:34.080983+0000","last_became_active":"2024-03-21T08:43:34.078610+0000","last_became_peered":"2024-03-21T08:43:34.078610+0000","last_unstale":"2024-03-21T08:43:34.080983+0000","last_undegraded":"2024-03-21T08:43:34.080983+0000","last_fullsized":"2024-03-21T08:43:34.080983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:32.786072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072634+0000","last_change":"2024-03-21T08:43:33.085060+0000","last_active":"2024-03-21T08:43:34.072634+0000","last_peered":"2024-03-21T08:43:34.072634+0000","last_clean":"2024-03-21T08:43:34.072634+0000","last_became_active":"2024-03-21T08:43:33.084725+0000","last_became_peered":"2024-03-21T08:43:33.084725+0000","last_unstale":"2024-03-21T08:43:34.072634+0000","last_undegraded":"2024-03-21T08:43:34.072634+0000","last_fullsized":"2024-03-21T08:43:34.072634+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063545+0000","last_change":"2024-03-21T08:43:33.063545+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063545+0000","last_undegraded":"2024-03-21T08:43:33.063545+0000","last_fullsized":"2024-03-21T08:43:33.063545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081061+0000","last_change":"2024-03-21T08:43:33.075978+0000","last_active":"2024-03-21T08:43:33.081061+0000","last_peered":"2024-03-21T08:43:33.081061+0000","last_clean":"2024-03-21T08:43:33.081061+0000","last_became_active":"2024-03-21T08:43:33.075230+0000","last_became_peered":"2024-03-21T08:43:33.075230+0000","last_unstale":"2024-03-21T08:43:33.081061+0000","last_undegraded":"2024-03-21T08:43:33.081061+0000","last_fullsized":"2024-03-21T08:43:33.081061+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.088395+0000","last_change":"2024-03-21T08:43:34.081971+0000","last_active":"2024-03-21T08:43:34.088395+0000","last_peered":"2024-03-21T08:43:34.088395+0000","last_clean":"2024-03-21T08:43:34.088395+0000","last_became_active":"2024-03-21T08:43:34.080960+0000","last_became_peered":"2024-03-21T08:43:34.080960+0000","last_unstale":"2024-03-21T08:43:34.088395+0000","last_undegraded":"2024-03-21T08:43:34.088395+0000","last_fullsized":"2024-03-21T08:43:34.088395+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:54.133660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"53'8","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.098747+0000","last_change":"2024-03-21T08:43:33.082199+0000","last_active":"2024-03-21T08:43:34.098747+0000","last_peered":"2024-03-21T08:43:34.098747+0000","last_clean":"2024-03-21T08:43:34.098747+0000","last_became_active":"2024-03-21T08:43:33.081892+0000","last_became_peered":"2024-03-21T08:43:33.081892+0000","last_unstale":"2024-03-21T08:43:34.098747+0000","last_undegraded":"2024-03-21T08:43:34.098747+0000","last_fullsized":"2024-03-21T08:43:34.098747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:19.722045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.092883+0000","last_change":"2024-03-21T08:43:34.086279+0000","last_active":"2024-03-21T08:43:34.092883+0000","last_peered":"2024-03-21T08:43:34.092883+0000","last_clean":"2024-03-21T08:43:34.092883+0000","last_became_active":"2024-03-21T08:43:34.086021+0000","last_became_peered":"2024-03-21T08:43:34.086021+0000","last_unstale":"2024-03-21T08:43:34.092883+0000","last_undegraded":"2024-03-21T08:43:34.092883+0000","last_fullsized":"2024-03-21T08:43:34.092883+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:51.667945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084347+0000","last_change":"2024-03-21T08:43:33.082173+0000","last_active":"2024-03-21T08:43:33.084347+0000","last_peered":"2024-03-21T08:43:33.084347+0000","last_clean":"2024-03-21T08:43:33.084347+0000","last_became_active":"2024-03-21T08:43:33.081902+0000","last_became_peered":"2024-03-21T08:43:33.081902+0000","last_unstale":"2024-03-21T08:43:33.084347+0000","last_undegraded":"2024-03-21T08:43:33.084347+0000","last_fullsized":"2024-03-21T08:43:33.084347+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.065237+0000","last_change":"2024-03-21T08:43:33.065237+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.065237+0000","last_undegraded":"2024-03-21T08:43:33.065237+0000","last_fullsized":"2024-03-21T08:43:33.065237+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083890+0000","last_change":"2024-03-21T08:43:33.077524+0000","last_active":"2024-03-21T08:43:33.083890+0000","last_peered":"2024-03-21T08:43:33.083890+0000","last_clean":"2024-03-21T08:43:33.083890+0000","last_became_active":"2024-03-21T08:43:33.077163+0000","last_became_peered":"2024-03-21T08:43:33.077163+0000","last_unstale":"2024-03-21T08:43:33.083890+0000","last_undegraded":"2024-03-21T08:43:33.083890+0000","last_fullsized":"2024-03-21T08:43:33.083890+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:39.382755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.069726+0000","last_change":"2024-03-21T08:43:33.069726+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.069726+0000","last_undegraded":"2024-03-21T08:43:33.069726+0000","last_fullsized":"2024-03-21T08:43:33.069726+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.083836+0000","last_change":"2024-03-21T08:43:33.078372+0000","last_active":"2024-03-21T08:43:33.083836+0000","last_peered":"2024-03-21T08:43:33.083836+0000","last_clean":"2024-03-21T08:43:33.083836+0000","last_became_active":"2024-03-21T08:43:33.077797+0000","last_became_peered":"2024-03-21T08:43:33.077797+0000","last_unstale":"2024-03-21T08:43:33.083836+0000","last_undegraded":"2024-03-21T08:43:33.083836+0000","last_fullsized":"2024-03-21T08:43:33.083836+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.076176+0000","last_change":"2024-03-21T08:43:34.072036+0000","last_active":"2024-03-21T08:43:34.076176+0000","last_peered":"2024-03-21T08:43:34.076176+0000","last_clean":"2024-03-21T08:43:34.076176+0000","last_became_active":"2024-03-21T08:43:34.071710+0000","last_became_peered":"2024-03-21T08:43:34.071710+0000","last_unstale":"2024-03-21T08:43:34.076176+0000","last_undegraded":"2024-03-21T08:43:34.076176+0000","last_fullsized":"2024-03-21T08:43:34.076176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:37.084279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084283+0000","last_change":"2024-03-21T08:43:33.079120+0000","last_active":"2024-03-21T08:43:33.084283+0000","last_peered":"2024-03-21T08:43:33.084283+0000","last_clean":"2024-03-21T08:43:33.084283+0000","last_became_active":"2024-03-21T08:43:33.078672+0000","last_became_peered":"2024-03-21T08:43:33.078672+0000","last_unstale":"2024-03-21T08:43:33.084283+0000","last_undegraded":"2024-03-21T08:43:33.084283+0000","last_fullsized":"2024-03-21T08:43:33.084283+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.068324+0000","last_change":"2024-03-21T08:43:33.068324+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.068324+0000","last_undegraded":"2024-03-21T08:43:33.068324+0000","last_fullsized":"2024-03-21T08:43:33.068324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076791+0000","last_change":"2024-03-21T08:43:33.074311+0000","last_active":"2024-03-21T08:43:33.076791+0000","last_peered":"2024-03-21T08:43:33.076791+0000","last_clean":"2024-03-21T08:43:33.076791+0000","last_became_active":"2024-03-21T08:43:33.074047+0000","last_became_peered":"2024-03-21T08:43:33.074047+0000","last_unstale":"2024-03-21T08:43:33.076791+0000","last_undegraded":"2024-03-21T08:43:33.076791+0000","last_fullsized":"2024-03-21T08:43:33.076791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062142+0000","last_change":"2024-03-21T08:43:33.062142+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062142+0000","last_undegraded":"2024-03-21T08:43:33.062142+0000","last_fullsized":"2024-03-21T08:43:33.062142+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.062005+0000","last_change":"2024-03-21T08:43:33.080008+0000","last_active":"2024-03-21T08:43:34.062005+0000","last_peered":"2024-03-21T08:43:34.062005+0000","last_clean":"2024-03-21T08:43:34.062005+0000","last_became_active":"2024-03-21T08:43:33.079744+0000","last_became_peered":"2024-03-21T08:43:33.079744+0000","last_unstale":"2024-03-21T08:43:34.062005+0000","last_undegraded":"2024-03-21T08:43:34.062005+0000","last_fullsized":"2024-03-21T08:43:34.062005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:34.225900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062049+0000","last_change":"2024-03-21T08:43:33.062049+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062049+0000","last_undegraded":"2024-03-21T08:43:33.062049+0000","last_fullsized":"2024-03-21T08:43:33.062049+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081118+0000","last_change":"2024-03-21T08:43:33.076251+0000","last_active":"2024-03-21T08:43:33.081118+0000","last_peered":"2024-03-21T08:43:33.081118+0000","last_clean":"2024-03-21T08:43:33.081118+0000","last_became_active":"2024-03-21T08:43:33.074661+0000","last_became_peered":"2024-03-21T08:43:33.074661+0000","last_unstale":"2024-03-21T08:43:33.081118+0000","last_undegraded":"2024-03-21T08:43:33.081118+0000","last_fullsized":"2024-03-21T08:43:33.081118+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:21.178601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064097+0000","last_change":"2024-03-21T08:43:33.064097+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064097+0000","last_undegraded":"2024-03-21T08:43:33.064097+0000","last_fullsized":"2024-03-21T08:43:33.064097+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"53'2","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.106185+0000","last_change":"2024-03-21T08:43:33.079309+0000","last_active":"2024-03-21T08:43:34.106185+0000","last_peered":"2024-03-21T08:43:34.106185+0000","last_clean":"2024-03-21T08:43:34.106185+0000","last_became_active":"2024-03-21T08:43:33.079038+0000","last_became_peered":"2024-03-21T08:43:33.079038+0000","last_unstale":"2024-03-21T08:43:34.106185+0000","last_undegraded":"2024-03-21T08:43:34.106185+0000","last_fullsized":"2024-03-21T08:43:34.106185+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:23.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064702+0000","last_change":"2024-03-21T08:43:33.064702+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064702+0000","last_undegraded":"2024-03-21T08:43:33.064702+0000","last_fullsized":"2024-03-21T08:43:33.064702+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084195+0000","last_change":"2024-03-21T08:43:33.078636+0000","last_active":"2024-03-21T08:43:33.084195+0000","last_peered":"2024-03-21T08:43:33.084195+0000","last_clean":"2024-03-21T08:43:33.084195+0000","last_became_active":"2024-03-21T08:43:33.078336+0000","last_became_peered":"2024-03-21T08:43:33.078336+0000","last_unstale":"2024-03-21T08:43:33.084195+0000","last_undegraded":"2024-03-21T08:43:33.084195+0000","last_fullsized":"2024-03-21T08:43:33.084195+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:40.803601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084289+0000","last_change":"2024-03-21T08:43:34.082469+0000","last_active":"2024-03-21T08:43:34.084289+0000","last_peered":"2024-03-21T08:43:34.084289+0000","last_clean":"2024-03-21T08:43:34.084289+0000","last_became_active":"2024-03-21T08:43:34.082227+0000","last_became_peered":"2024-03-21T08:43:34.082227+0000","last_unstale":"2024-03-21T08:43:34.084289+0000","last_undegraded":"2024-03-21T08:43:34.084289+0000","last_fullsized":"2024-03-21T08:43:34.084289+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:14.199100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.071019+0000","last_change":"2024-03-21T08:43:33.077725+0000","last_active":"2024-03-21T08:43:34.071019+0000","last_peered":"2024-03-21T08:43:34.071019+0000","last_clean":"2024-03-21T08:43:34.071019+0000","last_became_active":"2024-03-21T08:43:33.077452+0000","last_became_peered":"2024-03-21T08:43:33.077452+0000","last_unstale":"2024-03-21T08:43:34.071019+0000","last_undegraded":"2024-03-21T08:43:34.071019+0000","last_fullsized":"2024-03-21T08:43:34.071019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:22:33.956371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.064263+0000","last_change":"2024-03-21T08:43:33.064263+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.064263+0000","last_undegraded":"2024-03-21T08:43:33.064263+0000","last_fullsized":"2024-03-21T08:43:33.064263+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.081515+0000","last_change":"2024-03-21T08:43:33.076652+0000","last_active":"2024-03-21T08:43:33.081515+0000","last_peered":"2024-03-21T08:43:33.081515+0000","last_clean":"2024-03-21T08:43:33.081515+0000","last_became_active":"2024-03-21T08:43:33.074101+0000","last_became_peered":"2024-03-21T08:43:33.074101+0000","last_unstale":"2024-03-21T08:43:33.081515+0000","last_undegraded":"2024-03-21T08:43:33.081515+0000","last_fullsized":"2024-03-21T08:43:33.081515+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:41.699971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063671+0000","last_change":"2024-03-21T08:43:33.063671+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063671+0000","last_undegraded":"2024-03-21T08:43:33.063671+0000","last_fullsized":"2024-03-21T08:43:33.063671+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061812+0000","last_change":"2024-03-21T08:43:33.081506+0000","last_active":"2024-03-21T08:43:34.061812+0000","last_peered":"2024-03-21T08:43:34.061812+0000","last_clean":"2024-03-21T08:43:34.061812+0000","last_became_active":"2024-03-21T08:43:33.080894+0000","last_became_peered":"2024-03-21T08:43:33.080894+0000","last_unstale":"2024-03-21T08:43:34.061812+0000","last_undegraded":"2024-03-21T08:43:34.061812+0000","last_fullsized":"2024-03-21T08:43:34.061812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093878+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:34.093878+0000","last_peered":"2024-03-21T08:43:34.093878+0000","last_clean":"2024-03-21T08:43:34.093878+0000","last_became_active":"2024-03-21T08:43:34.089436+0000","last_became_peered":"2024-03-21T08:43:34.089436+0000","last_unstale":"2024-03-21T08:43:34.093878+0000","last_undegraded":"2024-03-21T08:43:34.093878+0000","last_fullsized":"2024-03-21T08:43:34.093878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:10.546830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.084246+0000","last_change":"2024-03-21T08:43:33.080132+0000","last_active":"2024-03-21T08:43:33.084246+0000","last_peered":"2024-03-21T08:43:33.084246+0000","last_clean":"2024-03-21T08:43:33.084246+0000","last_became_active":"2024-03-21T08:43:33.079867+0000","last_became_peered":"2024-03-21T08:43:33.079867+0000","last_unstale":"2024-03-21T08:43:33.084246+0000","last_undegraded":"2024-03-21T08:43:33.084246+0000","last_fullsized":"2024-03-21T08:43:33.084246+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:33:42.756681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.089682+0000","last_change":"2024-03-21T08:43:34.084715+0000","last_active":"2024-03-21T08:43:34.089682+0000","last_peered":"2024-03-21T08:43:34.089682+0000","last_clean":"2024-03-21T08:43:34.089682+0000","last_became_active":"2024-03-21T08:43:34.084436+0000","last_became_peered":"2024-03-21T08:43:34.084436+0000","last_unstale":"2024-03-21T08:43:34.089682+0000","last_undegraded":"2024-03-21T08:43:34.089682+0000","last_fullsized":"2024-03-21T08:43:34.089682+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:33:01.770928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061783+0000","last_change":"2024-03-21T08:43:33.076471+0000","last_active":"2024-03-21T08:43:34.061783+0000","last_peered":"2024-03-21T08:43:34.061783+0000","last_clean":"2024-03-21T08:43:34.061783+0000","last_became_active":"2024-03-21T08:43:33.076185+0000","last_became_peered":"2024-03-21T08:43:33.076185+0000","last_unstale":"2024-03-21T08:43:34.061783+0000","last_undegraded":"2024-03-21T08:43:34.061783+0000","last_fullsized":"2024-03-21T08:43:34.061783+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.062860+0000","last_change":"2024-03-21T08:43:33.062860+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.062860+0000","last_undegraded":"2024-03-21T08:43:33.062860+0000","last_fullsized":"2024-03-21T08:43:33.062860+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-21T08:43:33.076315+0000","last_change":"2024-03-21T08:43:33.074819+0000","last_active":"2024-03-21T08:43:33.076315+0000","last_peered":"2024-03-21T08:43:33.076315+0000","last_clean":"2024-03-21T08:43:33.076315+0000","last_became_active":"2024-03-21T08:43:33.074585+0000","last_became_peered":"2024-03-21T08:43:33.074585+0000","last_unstale":"2024-03-21T08:43:33.076315+0000","last_undegraded":"2024-03-21T08:43:33.076315+0000","last_fullsized":"2024-03-21T08:43:33.076315+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:20:49.617268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-21T08:43:33.063064+0000","last_change":"2024-03-21T08:43:33.063064+0000","last_active":"2024-03-21T08:43:33.047253+0000","last_peered":"2024-03-21T08:43:33.047253+0000","last_clean":"2024-03-21T08:43:33.047253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T08:43:33.063064+0000","last_undegraded":"2024-03-21T08:43:33.063064+0000","last_fullsized":"2024-03-21T08:43:33.063064+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072378+0000","last_change":"2024-03-21T08:43:33.081102+0000","last_active":"2024-03-21T08:43:34.072378+0000","last_peered":"2024-03-21T08:43:34.072378+0000","last_clean":"2024-03-21T08:43:34.072378+0000","last_became_active":"2024-03-21T08:43:33.080820+0000","last_became_peered":"2024-03-21T08:43:33.080820+0000","last_unstale":"2024-03-21T08:43:34.072378+0000","last_undegraded":"2024-03-21T08:43:34.072378+0000","last_fullsized":"2024-03-21T08:43:34.072378+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.065013+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:43:34.065013+0000","last_peered":"2024-03-21T08:43:34.065013+0000","last_clean":"2024-03-21T08:43:34.065013+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:43:34.065013+0000","last_undegraded":"2024-03-21T08:43:34.065013+0000","last_fullsized":"2024-03-21T08:43:34.065013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509476,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1228800,"data_stored":871737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":281209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":281299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542179,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":602112,"data_stored":267383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":864410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.066283+0000', 'last_change': '2024-03-21T08:43:33.066283+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.066283+0000', 'last_undegraded': '2024-03-21T08:43:33.066283+0000', 'last_fullsized': '2024-03-21T08:43:33.066283+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065481+0000', 'last_change': '2024-03-21T08:43:33.065481+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065481+0000', 'last_undegraded': '2024-03-21T08:43:33.065481+0000', 'last_fullsized': '2024-03-21T08:43:33.065481+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064693+0000', 'last_change': '2024-03-21T08:43:33.064693+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064693+0000', 'last_undegraded': '2024-03-21T08:43:33.064693+0000', 'last_fullsized': '2024-03-21T08:43:33.064693+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-03-21T08:43:38.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065768+0000', 'last_change': '2024-03-21T08:43:33.065768+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065768+0000', 'last_undegraded': '2024-03-21T08:43:33.065768+0000', 'last_fullsized': '2024-03-21T08:43:33.065768+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.062573+0000', 'last_change': '2024-03-21T08:43:33.062573+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.062573+0000', 'last_undegraded': '2024-03-21T08:43:33.062573+0000', 'last_fullsized': '2024-03-21T08:43:33.062573+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063862+0000', 'last_change': '2024-03-21T08:43:33.063862+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063862+0000', 'last_undegraded': '2024-03-21T08:43:33.063862+0000', 'last_fullsized': '2024-03-21T08:43:33.063862+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065944+0000', 'last_change': '2024-03-21T08:43:33.065944+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065944+0000', 'last_undegraded': '2024-03-21T08:43:33.065944+0000', 'last_fullsized': '2024-03-21T08:43:33.065944+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063655+0000', 'last_change': '2024-03-21T08:43:33.063655+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063655+0000', 'last_undegraded': '2024-03-21T08:43:33.063655+0000', 'last_fullsized': '2024-03-21T08:43:33.063655+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063371+0000', 'last_change': '2024-03-21T08:43:33.063371+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063371+0000', 'last_undegraded': '2024-03-21T08:43:33.063371+0000', 'last_fullsized': '2024-03-21T08:43:33.063371+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-21T08:43:38.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065030+0000', 'last_change': '2024-03-21T08:43:33.065030+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065030+0000', 'last_undegraded': '2024-03-21T08:43:33.065030+0000', 'last_fullsized': '2024-03-21T08:43:33.065030+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.068546+0000', 'last_change': '2024-03-21T08:43:33.068546+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.068546+0000', 'last_undegraded': '2024-03-21T08:43:33.068546+0000', 'last_fullsized': '2024-03-21T08:43:33.068546+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.070108+0000', 'last_change': '2024-03-21T08:43:33.070108+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.070108+0000', 'last_undegraded': '2024-03-21T08:43:33.070108+0000', 'last_fullsized': '2024-03-21T08:43:33.070108+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064406+0000', 'last_change': '2024-03-21T08:43:33.064406+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064406+0000', 'last_undegraded': '2024-03-21T08:43:33.064406+0000', 'last_fullsized': '2024-03-21T08:43:33.064406+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.069742+0000', 'last_change': '2024-03-21T08:43:33.069742+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.069742+0000', 'last_undegraded': '2024-03-21T08:43:33.069742+0000', 'last_fullsized': '2024-03-21T08:43:33.069742+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065512+0000', 'last_change': '2024-03-21T08:43:33.065512+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065512+0000', 'last_undegraded': '2024-03-21T08:43:33.065512+0000', 'last_fullsized': '2024-03-21T08:43:33.065512+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.068308+0000', 'last_change': '2024-03-21T08:43:33.068308+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.068308+0000', 'last_undegraded': '2024-03-21T08:43:33.068308+0000', 'last_fullsized': '2024-03-21T08:43:33.068308+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063487+0000', 'last_change': '2024-03-21T08:43:33.063487+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063487+0000', 'last_undegraded': '2024-03-21T08:43:33.063487+0000', 'last_fullsized': '2024-03-21T08:43:33.063487+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064212+0000', 'last_change': '2024-03-21T08:43:33.064212+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064212+0000', 'last_undegraded': '2024-03-21T08:43:33.064212+0000', 'last_fullsized': '2024-03-21T08:43:33.064212+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.062843+0000', 'last_change': '2024-03-21T08:43:33.062843+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.062843+0000', 'last_undegraded': '2024-03-21T08:43:33.062843+0000', 'last_fullsized': '2024-03-21T08:43:33.062843+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065020+0000', 'last_change': '2024-03-21T08:43:33.065020+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065020+0000', 'last_undegraded': '2024-03-21T08:43:33.065020+0000', 'last_fullsized': '2024-03-21T08:43:33.065020+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064047+0000', 'last_change': '2024-03-21T08:43:33.064047+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064047+0000', 'last_undegraded': '2024-03-21T08:43:33.064047+0000', 'last_fullsized': '2024-03-21T08:43:33.064047+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.069118+0000', 'last_change': '2024-03-21T08:43:33.069118+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.069118+0000', 'last_undegraded': '2024-03-21T08:43:33.069118+0000', 'last_fullsized': '2024-03-21T08:43:33.069118+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063254+0000', 'last_change': '2024-03-21T08:43:33.063254+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063254+0000', 'last_undegraded': '2024-03-21T08:43:33.063254+0000', 'last_fullsized': '2024-03-21T08:43:33.063254+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063758+0000', 'last_change': '2024-03-21T08:43:33.063758+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063758+0000', 'last_undegraded': '2024-03-21T08:43:33.063758+0000', 'last_fullsized': '2024-03-21T08:43:33.063758+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.068759+0000', 'last_change': '2024-03-21T08:43:33.068759+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.068759+0000', 'last_undegraded': '2024-03-21T08:43:33.068759+0000', 'last_fullsized': '2024-03-21T08:43:33.068759+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063545+0000', 'last_change': '2024-03-21T08:43:33.063545+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063545+0000', 'last_undegraded': '2024-03-21T08:43:33.063545+0000', 'last_fullsized': '2024-03-21T08:43:33.063545+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-21T08:43:38.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.065237+0000', 'last_change': '2024-03-21T08:43:33.065237+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.065237+0000', 'last_undegraded': '2024-03-21T08:43:33.065237+0000', 'last_fullsized': '2024-03-21T08:43:33.065237+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.069726+0000', 'last_change': '2024-03-21T08:43:33.069726+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.069726+0000', 'last_undegraded': '2024-03-21T08:43:33.069726+0000', 'last_fullsized': '2024-03-21T08:43:33.069726+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.068324+0000', 'last_change': '2024-03-21T08:43:33.068324+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.068324+0000', 'last_undegraded': '2024-03-21T08:43:33.068324+0000', 'last_fullsized': '2024-03-21T08:43:33.068324+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.062142+0000', 'last_change': '2024-03-21T08:43:33.062142+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.062142+0000', 'last_undegraded': '2024-03-21T08:43:33.062142+0000', 'last_fullsized': '2024-03-21T08:43:33.062142+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.062049+0000', 'last_change': '2024-03-21T08:43:33.062049+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.062049+0000', 'last_undegraded': '2024-03-21T08:43:33.062049+0000', 'last_fullsized': '2024-03-21T08:43:33.062049+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064097+0000', 'last_change': '2024-03-21T08:43:33.064097+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064097+0000', 'last_undegraded': '2024-03-21T08:43:33.064097+0000', 'last_fullsized': '2024-03-21T08:43:33.064097+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064702+0000', 'last_change': '2024-03-21T08:43:33.064702+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064702+0000', 'last_undegraded': '2024-03-21T08:43:33.064702+0000', 'last_fullsized': '2024-03-21T08:43:33.064702+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.064263+0000', 'last_change': '2024-03-21T08:43:33.064263+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.064263+0000', 'last_undegraded': '2024-03-21T08:43:33.064263+0000', 'last_fullsized': '2024-03-21T08:43:33.064263+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063671+0000', 'last_change': '2024-03-21T08:43:33.063671+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063671+0000', 'last_undegraded': '2024-03-21T08:43:33.063671+0000', 'last_fullsized': '2024-03-21T08:43:33.063671+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.062860+0000', 'last_change': '2024-03-21T08:43:33.062860+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.062860+0000', 'last_undegraded': '2024-03-21T08:43:33.062860+0000', 'last_fullsized': '2024-03-21T08:43:33.062860+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-03-21T08:43:38.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-03-21T08:43:33.063064+0000', 'last_change': '2024-03-21T08:43:33.063064+0000', 'last_active': '2024-03-21T08:43:33.047253+0000', 'last_peered': '2024-03-21T08:43:33.047253+0000', 'last_clean': '2024-03-21T08:43:33.047253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-21T08:43:33.063064+0000', 'last_undegraded': '2024-03-21T08:43:33.063064+0000', 'last_fullsized': '2024-03-21T08:43:33.063064+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'last_clean_scrub_stamp': '2024-03-21T08:43:33.047253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-21T08:43:38.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-21T08:43:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.154+0000 7fcb72000640 1 Processor -- start 2024-03-21T08:43:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.154+0000 7fcb72000640 1 -- start start 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb72000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 0x7fcb6c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb72000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c15ebf0 con 0x7fcb6c15e610 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb72000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c15eed0 con 0x7fcb6c0ce490 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb72000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c15f1b0 con 0x7fcb6c15ba70 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcb6c15e610 0x7fcb6c0b4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38698/0 (socket says 172.21.15.28:38698) 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6aa00640 1 -- 172.21.15.28:0/2598458627 learned_addr learned my addr 172.21.15.28:0/2598458627 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6be00640 1 --2- 172.21.15.28:0/2598458627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 0x7fcb6c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271515602 0 0) 0x7fcb6c15ebf0 con 0x7fcb6c15e610 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb50003530 con 0x7fcb6c15e610 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547863452 0 0) 0x7fcb6c15eed0 con 0x7fcb6c0ce490 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c15ebf0 con 0x7fcb6c0ce490 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6be00640 1 -- 172.21.15.28:0/2598458627 >> v1:172.21.15.151:6789/0 conn(0x7fcb6c0ce490 legacy=0x7fcb6c0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6be00640 1 -- 172.21.15.28:0/2598458627 >> v1:172.21.15.28:6789/0 conn(0x7fcb6c15e610 legacy=0x7fcb6c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.158+0000 7fcb6be00640 1 -- 172.21.15.28:0/2598458627 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c15f490 con 0x7fcb6c15ba70 2024-03-21T08:43:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb6be00640 1 --2- 172.21.15.28:0/2598458627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 0x7fcb6c0b4a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009dc0 tx=0x7fcb6401c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b9dc093955f25a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb640b2070 con 0x7fcb6c15ba70 2024-03-21T08:43:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb640a57c0 con 0x7fcb6c15ba70 2024-03-21T08:43:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb6a000640 1 -- 172.21.15.28:0/2598458627 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb640a5b90 con 0x7fcb6c15ba70 2024-03-21T08:43:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- 172.21.15.28:0/2598458627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 msgr2=0x7fcb6c0b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 --2- 172.21.15.28:0/2598458627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 0x7fcb6c0b4a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009dc0 tx=0x7fcb6401c900 comp rx=0 tx=0).stop 2024-03-21T08:43:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- 172.21.15.28:0/2598458627 shutdown_connections 2024-03-21T08:43:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 --2- 172.21.15.28:0/2598458627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c15ba70 0x7fcb6c0b4a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- 172.21.15.28:0/2598458627 >> 172.21.15.28:0/2598458627 conn(0x7fcb6c0cc230 msgr2=0x7fcb6c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- 172.21.15.28:0/2598458627 shutdown_connections 2024-03-21T08:43:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- 172.21.15.28:0/2598458627 wait complete. 2024-03-21T08:43:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 Processor -- start 2024-03-21T08:43:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.162+0000 7fcb72000640 1 -- start start 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c0ce490 0x7fcb6c15df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb6c15e610 0x7fcb6c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c168b40 con 0x7fcb6c15ba70 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1689c0 con 0x7fcb6c15e610 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c168cc0 con 0x7fcb6c0ce490 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43488/0 (socket says 172.21.15.28:43488) 2024-03-21T08:43:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 -- 172.21.15.28:0/2471251842 learned_addr learned my addr 172.21.15.28:0/2471251842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6b400640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c0ce490 0x7fcb6c15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6be00640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb6c15e610 0x7fcb6c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 -- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c0ce490 msgr2=0x7fcb6c15df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c0ce490 0x7fcb6c15df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 -- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb6c15e610 msgr2=0x7fcb6c0afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb6c15e610 0x7fcb6c0afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 -- 172.21.15.28:0/2471251842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c146a00 con 0x7fcb6c15ba70 2024-03-21T08:43:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb6aa00640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c440 tx=0x7fcb5800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58017070 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58005cf0 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c14a700 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007290 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c14acc0 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c04f410 con 0x7fcb6c15ba70 2024-03-21T08:43:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.166+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fcb5801a070 con 0x7fcb6c15ba70 2024-03-21T08:43:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.170+0000 7fcb63400640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 0x7fcb4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.170+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb58085480 con 0x7fcb6c15ba70 2024-03-21T08:43:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.170+0000 7fcb6b400640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 0x7fcb4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.170+0000 7fcb6b400640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 0x7fcb4c069fe0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0011f0 tx=0x7fcb5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.174+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcb58055c60 con 0x7fcb6c15ba70 2024-03-21T08:43:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.390+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fcb6c0cbfa0 con 0x7fcb6c15ba70 2024-03-21T08:43:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb63400640 1 -- 172.21.15.28:0/2471251842 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1258 (secure 0 0 0) 0x7fcb58055560 con 0x7fcb6c15ba70 2024-03-21T08:43:39.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:39.395 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"51dba44a-cba1-4b57-a836-26a407968834","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":176,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1711010453,"num_in_osds":8,"osd_in_since":1711010445,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":60},{"state_name":"creating+peering","count":37}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":590458,"bytes_used":244543488,"bytes_avail":772849569792,"bytes_total":773094113280,"inactive_pgs_ratio":0.38144329190254211,"write_bytes_sec":179,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":28,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":6249}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-21T08:43:25.751084+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 msgr2=0x7fcb4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 0x7fcb4c069fe0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0011f0 tx=0x7fcb5c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 msgr2=0x7fcb6c1464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c440 tx=0x7fcb5800c900 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 shutdown_connections 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcb6c15e610 0x7fcb6c0afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6c15ba70 0x7fcb6c1464c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcb4c067b20 0x7fcb4c069fe0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 --2- 172.21.15.28:0/2471251842 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcb6c0ce490 0x7fcb6c15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 >> 172.21.15.28:0/2471251842 conn(0x7fcb6c0cc230 msgr2=0x7fcb6c15a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 shutdown_connections 2024-03-21T08:43:39.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.394+0000 7fcb72000640 1 -- 172.21.15.28:0/2471251842 wait complete. 2024-03-21T08:43:39.421 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-21T08:43:39.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:43:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.558+0000 7f97b5a00640 1 Processor -- start 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.558+0000 7f97b5a00640 1 -- start start 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97b5a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f97b0099b80 0x7f97b009a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97b5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97b00cfcb0 0x7f97b00d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97b5a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b00cc1d0 con 0x7f97b015e0c0 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97b5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b00d1730 con 0x7f97b00cfcb0 2024-03-21T08:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97b5a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b00d1450 con 0x7f97b0099b80 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97b00cfcb0 0x7f97b00d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97b00cfcb0 0x7f97b00d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43496/0 (socket says 172.21.15.28:43496) 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 -- 172.21.15.28:0/2451943210 learned_addr learned my addr 172.21.15.28:0/2451943210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97a7e00640 1 --2- 172.21.15.28:0/2451943210 >> v2:172.21.15.151:3300/0 conn(0x7f97b0099b80 0x7f97b009a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 -- 172.21.15.28:0/2451943210 >> v2:172.21.15.151:3300/0 conn(0x7f97b0099b80 msgr2=0x7f97b009a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 --2- 172.21.15.28:0/2451943210 >> v2:172.21.15.151:3300/0 conn(0x7f97b0099b80 0x7f97b009a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 -- 172.21.15.28:0/2451943210 >> v1:172.21.15.151:6789/0 conn(0x7f97b015e0c0 legacy=0x7f97b006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.562+0000 7f97af400640 1 -- 172.21.15.28:0/2451943210 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00cc470 con 0x7f97b00cfcb0 2024-03-21T08:43:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97af400640 1 --2- 172.21.15.28:0/2451943210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b00cfcb0 0x7f97b00d0120 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f97b0072ea0 tx=0x7f97a801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c2ffc7b544b07e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97ae000640 1 -- 172.21.15.28:0/2451943210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a80a5740 con 0x7f97b00cfcb0 2024-03-21T08:43:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97ae000640 1 -- 172.21.15.28:0/2451943210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a80a58e0 con 0x7f97b00cfcb0 2024-03-21T08:43:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97ae000640 1 -- 172.21.15.28:0/2451943210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a80a5cb0 con 0x7f97b00cfcb0 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- 172.21.15.28:0/2451943210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b00cfcb0 msgr2=0x7f97b00d0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 --2- 172.21.15.28:0/2451943210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b00cfcb0 0x7f97b00d0120 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f97b0072ea0 tx=0x7f97a801c9f0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- 172.21.15.28:0/2451943210 shutdown_connections 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 --2- 172.21.15.28:0/2451943210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b00cfcb0 0x7f97b00d0120 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 --2- 172.21.15.28:0/2451943210 >> v2:172.21.15.151:3300/0 conn(0x7f97b0099b80 0x7f97b009a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- 172.21.15.28:0/2451943210 >> 172.21.15.28:0/2451943210 conn(0x7f97b00b34d0 msgr2=0x7f97b00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- 172.21.15.28:0/2451943210 shutdown_connections 2024-03-21T08:43:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- 172.21.15.28:0/2451943210 wait complete. 2024-03-21T08:43:39.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 Processor -- start 2024-03-21T08:43:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.566+0000 7f97b5a00640 1 -- start start 2024-03-21T08:43:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97b00cfcb0 0x7f97b00c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97b015e0c0 0x7f97b014b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b00d0e70 con 0x7f97b0099b80 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b00d0ff0 con 0x7f97b00cfcb0 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b00d1170 con 0x7f97b015e0c0 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43502/0 (socket says 172.21.15.28:43502) 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 -- 172.21.15.28:0/1374219454 learned_addr learned my addr 172.21.15.28:0/1374219454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97b00cfcb0 0x7f97b00c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97af400640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97b015e0c0 0x7f97b014b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 -- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97b015e0c0 msgr2=0x7f97b014b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97b015e0c0 0x7f97b014b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 -- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97b00cfcb0 msgr2=0x7f97b00c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97b00cfcb0 0x7f97b00c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 -- 172.21.15.28:0/1374219454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00c5eb0 con 0x7f97b0099b80 2024-03-21T08:43:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97aea00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f97a000c450 tx=0x7f97a000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0016020 con 0x7f97b0099b80 2024-03-21T08:43:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a00052d0 con 0x7f97b0099b80 2024-03-21T08:43:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b00c6190 con 0x7f97b0099b80 2024-03-21T08:43:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0007260 con 0x7f97b0099b80 2024-03-21T08:43:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b015a740 con 0x7f97b0099b80 2024-03-21T08:43:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b004f410 con 0x7f97b0099b80 2024-03-21T08:43:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.570+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f97a0007400 con 0x7f97b0099b80 2024-03-21T08:43:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.574+0000 7f97a6a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 0x7f978c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.574+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97a0085920 con 0x7f97b0099b80 2024-03-21T08:43:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.574+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 0x7f978c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.574+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 0x7f978c069fe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f97b00b71b0 tx=0x7f9798009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.578+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f97a0020020 con 0x7f97b0099b80 2024-03-21T08:43:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.846+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97b006d050 con 0x7f978c067b20 2024-03-21T08:43:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.854+0000 7f97a6a00640 1 -- 172.21.15.28:0/1374219454 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251927 (secure 0 0 0) 0x7f97b006d050 con 0x7f978c067b20 2024-03-21T08:43:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:43:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 msgr2=0x7f978c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 0x7f978c069fe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f97b00b71b0 tx=0x7f9798009190 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 msgr2=0x7f97b014d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f97a000c450 tx=0x7f97a000c910 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 shutdown_connections 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f978c067b20 0x7f978c069fe0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f97b015e0c0 0x7f97b014b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f97b00cfcb0 0x7f97b00c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 --2- 172.21.15.28:0/1374219454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b0099b80 0x7f97b014d850 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 >> 172.21.15.28:0/1374219454 conn(0x7f97b00b34d0 msgr2=0x7f97b006fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 shutdown_connections 2024-03-21T08:43:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:39.858+0000 7f97b5a00640 1 -- 172.21.15.28:0/1374219454 wait complete. 2024-03-21T08:43:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-03-21T08:43:39.754054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238968,"kb_used_data":6148,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735752,"statfs":{"total":773094113280,"available":772849410048,"internally_reserved":0,"allocated":6295552,"data_stored":3454773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.694159"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.105337+0000","last_change":"2024-03-21T08:43:34.104498+0000","last_active":"2024-03-21T08:43:34.105337+0000","last_peered":"2024-03-21T08:43:34.105337+0000","last_clean":"2024-03-21T08:43:34.105337+0000","last_became_active":"2024-03-21T08:43:34.102336+0000","last_became_peered":"2024-03-21T08:43:34.102336+0000","last_unstale":"2024-03-21T08:43:34.105337+0000","last_undegraded":"2024-03-21T08:43:34.105337+0000","last_fullsized":"2024-03-21T08:43:34.105337+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:44:57.769559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267179+0000","last_change":"2024-03-21T08:43:34.088311+0000","last_active":"2024-03-21T08:43:35.267179+0000","last_peered":"2024-03-21T08:43:35.267179+0000","last_clean":"2024-03-21T08:43:35.267179+0000","last_became_active":"2024-03-21T08:43:34.087919+0000","last_became_peered":"2024-03-21T08:43:34.087919+0000","last_unstale":"2024-03-21T08:43:35.267179+0000","last_undegraded":"2024-03-21T08:43:35.267179+0000","last_fullsized":"2024-03-21T08:43:35.267179+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:32:01.332600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084512+0000","last_change":"2024-03-21T08:43:34.079798+0000","last_active":"2024-03-21T08:43:34.084512+0000","last_peered":"2024-03-21T08:43:34.084512+0000","last_clean":"2024-03-21T08:43:34.084512+0000","last_became_active":"2024-03-21T08:43:34.079557+0000","last_became_peered":"2024-03-21T08:43:34.079557+0000","last_unstale":"2024-03-21T08:43:34.084512+0000","last_undegraded":"2024-03-21T08:43:34.084512+0000","last_fullsized":"2024-03-21T08:43:34.084512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:33:45.824524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065349+0000","last_change":"2024-03-21T08:43:34.069789+0000","last_active":"2024-03-21T08:43:35.065349+0000","last_peered":"2024-03-21T08:43:35.065349+0000","last_clean":"2024-03-21T08:43:35.065349+0000","last_became_active":"2024-03-21T08:43:34.068694+0000","last_became_peered":"2024-03-21T08:43:34.068694+0000","last_unstale":"2024-03-21T08:43:35.065349+0000","last_undegraded":"2024-03-21T08:43:35.065349+0000","last_fullsized":"2024-03-21T08:43:35.065349+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:20:04.245491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074891+0000","last_change":"2024-03-21T08:43:34.069862+0000","last_active":"2024-03-21T08:43:34.074891+0000","last_peered":"2024-03-21T08:43:34.074891+0000","last_clean":"2024-03-21T08:43:34.074891+0000","last_became_active":"2024-03-21T08:43:34.069561+0000","last_became_peered":"2024-03-21T08:43:34.069561+0000","last_unstale":"2024-03-21T08:43:34.074891+0000","last_undegraded":"2024-03-21T08:43:34.074891+0000","last_fullsized":"2024-03-21T08:43:34.074891+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:02:24.282774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083581+0000","last_change":"2024-03-21T08:43:34.104956+0000","last_active":"2024-03-21T08:43:35.083581+0000","last_peered":"2024-03-21T08:43:35.083581+0000","last_clean":"2024-03-21T08:43:35.083581+0000","last_became_active":"2024-03-21T08:43:34.104667+0000","last_became_peered":"2024-03-21T08:43:34.104667+0000","last_unstale":"2024-03-21T08:43:35.083581+0000","last_undegraded":"2024-03-21T08:43:35.083581+0000","last_fullsized":"2024-03-21T08:43:35.083581+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:52.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265410+0000","last_change":"2024-03-21T08:43:34.086151+0000","last_active":"2024-03-21T08:43:35.265410+0000","last_peered":"2024-03-21T08:43:35.265410+0000","last_clean":"2024-03-21T08:43:35.265410+0000","last_became_active":"2024-03-21T08:43:34.085809+0000","last_became_peered":"2024-03-21T08:43:34.085809+0000","last_unstale":"2024-03-21T08:43:35.265410+0000","last_undegraded":"2024-03-21T08:43:35.265410+0000","last_fullsized":"2024-03-21T08:43:35.265410+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:40.388982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266351+0000","last_change":"2024-03-21T08:43:34.083980+0000","last_active":"2024-03-21T08:43:35.266351+0000","last_peered":"2024-03-21T08:43:35.266351+0000","last_clean":"2024-03-21T08:43:35.266351+0000","last_became_active":"2024-03-21T08:43:34.083638+0000","last_became_peered":"2024-03-21T08:43:34.083638+0000","last_unstale":"2024-03-21T08:43:35.266351+0000","last_undegraded":"2024-03-21T08:43:35.266351+0000","last_fullsized":"2024-03-21T08:43:35.266351+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:28:55.959297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265818+0000","last_change":"2024-03-21T08:43:34.090630+0000","last_active":"2024-03-21T08:43:35.265818+0000","last_peered":"2024-03-21T08:43:35.265818+0000","last_clean":"2024-03-21T08:43:35.265818+0000","last_became_active":"2024-03-21T08:43:34.090323+0000","last_became_peered":"2024-03-21T08:43:34.090323+0000","last_unstale":"2024-03-21T08:43:35.265818+0000","last_undegraded":"2024-03-21T08:43:35.265818+0000","last_fullsized":"2024-03-21T08:43:35.265818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:22.184827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079493+0000","last_change":"2024-03-21T08:43:34.076812+0000","last_active":"2024-03-21T08:43:34.079493+0000","last_peered":"2024-03-21T08:43:34.079493+0000","last_clean":"2024-03-21T08:43:34.079493+0000","last_became_active":"2024-03-21T08:43:34.076556+0000","last_became_peered":"2024-03-21T08:43:34.076556+0000","last_unstale":"2024-03-21T08:43:34.079493+0000","last_undegraded":"2024-03-21T08:43:34.079493+0000","last_fullsized":"2024-03-21T08:43:34.079493+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:17:35.654015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.099722+0000","last_change":"2024-03-21T08:43:34.096375+0000","last_active":"2024-03-21T08:43:34.099722+0000","last_peered":"2024-03-21T08:43:34.099722+0000","last_clean":"2024-03-21T08:43:34.099722+0000","last_became_active":"2024-03-21T08:43:34.095943+0000","last_became_peered":"2024-03-21T08:43:34.095943+0000","last_unstale":"2024-03-21T08:43:34.099722+0000","last_undegraded":"2024-03-21T08:43:34.099722+0000","last_fullsized":"2024-03-21T08:43:34.099722+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:20.728271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065120+0000","last_change":"2024-03-21T08:43:34.075290+0000","last_active":"2024-03-21T08:43:35.065120+0000","last_peered":"2024-03-21T08:43:35.065120+0000","last_clean":"2024-03-21T08:43:35.065120+0000","last_became_active":"2024-03-21T08:43:34.074620+0000","last_became_peered":"2024-03-21T08:43:34.074620+0000","last_unstale":"2024-03-21T08:43:35.065120+0000","last_undegraded":"2024-03-21T08:43:35.065120+0000","last_fullsized":"2024-03-21T08:43:35.065120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:20:04.099452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094467+0000","last_change":"2024-03-21T08:43:34.090738+0000","last_active":"2024-03-21T08:43:34.094467+0000","last_peered":"2024-03-21T08:43:34.094467+0000","last_clean":"2024-03-21T08:43:34.094467+0000","last_became_active":"2024-03-21T08:43:34.090493+0000","last_became_peered":"2024-03-21T08:43:34.090493+0000","last_unstale":"2024-03-21T08:43:34.094467+0000","last_undegraded":"2024-03-21T08:43:34.094467+0000","last_fullsized":"2024-03-21T08:43:34.094467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265408+0000","last_change":"2024-03-21T08:43:34.101001+0000","last_active":"2024-03-21T08:43:35.265408+0000","last_peered":"2024-03-21T08:43:35.265408+0000","last_clean":"2024-03-21T08:43:35.265408+0000","last_became_active":"2024-03-21T08:43:34.100713+0000","last_became_peered":"2024-03-21T08:43:34.100713+0000","last_unstale":"2024-03-21T08:43:35.265408+0000","last_undegraded":"2024-03-21T08:43:35.265408+0000","last_fullsized":"2024-03-21T08:43:35.265408+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:56:36.030152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265532+0000","last_change":"2024-03-21T08:43:34.083354+0000","last_active":"2024-03-21T08:43:35.265532+0000","last_peered":"2024-03-21T08:43:35.265532+0000","last_clean":"2024-03-21T08:43:35.265532+0000","last_became_active":"2024-03-21T08:43:34.082948+0000","last_became_peered":"2024-03-21T08:43:34.082948+0000","last_unstale":"2024-03-21T08:43:35.265532+0000","last_undegraded":"2024-03-21T08:43:35.265532+0000","last_fullsized":"2024-03-21T08:43:35.265532+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:15:32.884893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083484+0000","last_change":"2024-03-21T08:43:34.088026+0000","last_active":"2024-03-21T08:43:35.083484+0000","last_peered":"2024-03-21T08:43:35.083484+0000","last_clean":"2024-03-21T08:43:35.083484+0000","last_became_active":"2024-03-21T08:43:34.087692+0000","last_became_peered":"2024-03-21T08:43:34.087692+0000","last_unstale":"2024-03-21T08:43:35.083484+0000","last_undegraded":"2024-03-21T08:43:35.083484+0000","last_fullsized":"2024-03-21T08:43:35.083484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:25.666989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267370+0000","last_change":"2024-03-21T08:43:34.098356+0000","last_active":"2024-03-21T08:43:35.267370+0000","last_peered":"2024-03-21T08:43:35.267370+0000","last_clean":"2024-03-21T08:43:35.267370+0000","last_became_active":"2024-03-21T08:43:34.098090+0000","last_became_peered":"2024-03-21T08:43:34.098090+0000","last_unstale":"2024-03-21T08:43:35.267370+0000","last_undegraded":"2024-03-21T08:43:35.267370+0000","last_fullsized":"2024-03-21T08:43:35.267370+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:55.221677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093933+0000","last_change":"2024-03-21T08:43:34.090757+0000","last_active":"2024-03-21T08:43:34.093933+0000","last_peered":"2024-03-21T08:43:34.093933+0000","last_clean":"2024-03-21T08:43:34.093933+0000","last_became_active":"2024-03-21T08:43:34.090125+0000","last_became_peered":"2024-03-21T08:43:34.090125+0000","last_unstale":"2024-03-21T08:43:34.093933+0000","last_undegraded":"2024-03-21T08:43:34.093933+0000","last_fullsized":"2024-03-21T08:43:34.093933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:14:11.321572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.103454+0000","last_change":"2024-03-21T08:43:34.099533+0000","last_active":"2024-03-21T08:43:34.103454+0000","last_peered":"2024-03-21T08:43:34.103454+0000","last_clean":"2024-03-21T08:43:34.103454+0000","last_became_active":"2024-03-21T08:43:34.099273+0000","last_became_peered":"2024-03-21T08:43:34.099273+0000","last_unstale":"2024-03-21T08:43:34.103454+0000","last_undegraded":"2024-03-21T08:43:34.103454+0000","last_fullsized":"2024-03-21T08:43:34.103454+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:25:59.522961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267033+0000","last_change":"2024-03-21T08:43:34.097269+0000","last_active":"2024-03-21T08:43:35.267033+0000","last_peered":"2024-03-21T08:43:35.267033+0000","last_clean":"2024-03-21T08:43:35.267033+0000","last_became_active":"2024-03-21T08:43:34.096445+0000","last_became_peered":"2024-03-21T08:43:34.096445+0000","last_unstale":"2024-03-21T08:43:35.267033+0000","last_undegraded":"2024-03-21T08:43:35.267033+0000","last_fullsized":"2024-03-21T08:43:35.267033+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:30:29.699066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073021+0000","last_change":"2024-03-21T08:43:34.069515+0000","last_active":"2024-03-21T08:43:34.073021+0000","last_peered":"2024-03-21T08:43:34.073021+0000","last_clean":"2024-03-21T08:43:34.073021+0000","last_became_active":"2024-03-21T08:43:34.069239+0000","last_became_peered":"2024-03-21T08:43:34.069239+0000","last_unstale":"2024-03-21T08:43:34.073021+0000","last_undegraded":"2024-03-21T08:43:34.073021+0000","last_fullsized":"2024-03-21T08:43:34.073021+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:40:59.946986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266593+0000","last_change":"2024-03-21T08:43:34.101974+0000","last_active":"2024-03-21T08:43:35.266593+0000","last_peered":"2024-03-21T08:43:35.266593+0000","last_clean":"2024-03-21T08:43:35.266593+0000","last_became_active":"2024-03-21T08:43:34.101668+0000","last_became_peered":"2024-03-21T08:43:34.101668+0000","last_unstale":"2024-03-21T08:43:35.266593+0000","last_undegraded":"2024-03-21T08:43:35.266593+0000","last_fullsized":"2024-03-21T08:43:35.266593+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:57:05.618262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100244+0000","last_change":"2024-03-21T08:43:34.097093+0000","last_active":"2024-03-21T08:43:34.100244+0000","last_peered":"2024-03-21T08:43:34.100244+0000","last_clean":"2024-03-21T08:43:34.100244+0000","last_became_active":"2024-03-21T08:43:34.096448+0000","last_became_peered":"2024-03-21T08:43:34.096448+0000","last_unstale":"2024-03-21T08:43:34.100244+0000","last_undegraded":"2024-03-21T08:43:34.100244+0000","last_fullsized":"2024-03-21T08:43:34.100244+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:20:52.859039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.094188+0000","last_change":"2024-03-21T08:43:34.089864+0000","last_active":"2024-03-21T08:43:34.094188+0000","last_peered":"2024-03-21T08:43:34.094188+0000","last_clean":"2024-03-21T08:43:34.094188+0000","last_became_active":"2024-03-21T08:43:34.089605+0000","last_became_peered":"2024-03-21T08:43:34.089605+0000","last_unstale":"2024-03-21T08:43:34.094188+0000","last_undegraded":"2024-03-21T08:43:34.094188+0000","last_fullsized":"2024-03-21T08:43:34.094188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:24.367969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074927+0000","last_change":"2024-03-21T08:43:34.070708+0000","last_active":"2024-03-21T08:43:34.074927+0000","last_peered":"2024-03-21T08:43:34.074927+0000","last_clean":"2024-03-21T08:43:34.074927+0000","last_became_active":"2024-03-21T08:43:34.070458+0000","last_became_peered":"2024-03-21T08:43:34.070458+0000","last_unstale":"2024-03-21T08:43:34.074927+0000","last_undegraded":"2024-03-21T08:43:34.074927+0000","last_fullsized":"2024-03-21T08:43:34.074927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:23:48.402969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065136+0000","last_change":"2024-03-21T08:43:34.084818+0000","last_active":"2024-03-21T08:43:35.065136+0000","last_peered":"2024-03-21T08:43:35.065136+0000","last_clean":"2024-03-21T08:43:35.065136+0000","last_became_active":"2024-03-21T08:43:34.084490+0000","last_became_peered":"2024-03-21T08:43:34.084490+0000","last_unstale":"2024-03-21T08:43:35.065136+0000","last_undegraded":"2024-03-21T08:43:35.065136+0000","last_fullsized":"2024-03-21T08:43:35.065136+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:13:14.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083917+0000","last_change":"2024-03-21T08:43:34.082575+0000","last_active":"2024-03-21T08:43:35.083917+0000","last_peered":"2024-03-21T08:43:35.083917+0000","last_clean":"2024-03-21T08:43:35.083917+0000","last_became_active":"2024-03-21T08:43:34.082126+0000","last_became_peered":"2024-03-21T08:43:34.082126+0000","last_unstale":"2024-03-21T08:43:35.083917+0000","last_undegraded":"2024-03-21T08:43:35.083917+0000","last_fullsized":"2024-03-21T08:43:35.083917+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:03.252966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080452+0000","last_change":"2024-03-21T08:43:34.074171+0000","last_active":"2024-03-21T08:43:34.080452+0000","last_peered":"2024-03-21T08:43:34.080452+0000","last_clean":"2024-03-21T08:43:34.080452+0000","last_became_active":"2024-03-21T08:43:34.073918+0000","last_became_peered":"2024-03-21T08:43:34.073918+0000","last_unstale":"2024-03-21T08:43:34.080452+0000","last_undegraded":"2024-03-21T08:43:34.080452+0000","last_fullsized":"2024-03-21T08:43:34.080452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:23:54.394841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266980+0000","last_change":"2024-03-21T08:43:34.086013+0000","last_active":"2024-03-21T08:43:35.266980+0000","last_peered":"2024-03-21T08:43:35.266980+0000","last_clean":"2024-03-21T08:43:35.266980+0000","last_became_active":"2024-03-21T08:43:34.082821+0000","last_became_peered":"2024-03-21T08:43:34.082821+0000","last_unstale":"2024-03-21T08:43:35.266980+0000","last_undegraded":"2024-03-21T08:43:35.266980+0000","last_fullsized":"2024-03-21T08:43:35.266980+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:44:13.212743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.085472+0000","last_change":"2024-03-21T08:43:34.079738+0000","last_active":"2024-03-21T08:43:34.085472+0000","last_peered":"2024-03-21T08:43:34.085472+0000","last_clean":"2024-03-21T08:43:34.085472+0000","last_became_active":"2024-03-21T08:43:34.079479+0000","last_became_peered":"2024-03-21T08:43:34.079479+0000","last_unstale":"2024-03-21T08:43:34.085472+0000","last_undegraded":"2024-03-21T08:43:34.085472+0000","last_fullsized":"2024-03-21T08:43:34.085472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:22:25.192237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.073626+0000","last_change":"2024-03-21T08:43:34.069752+0000","last_active":"2024-03-21T08:43:34.073626+0000","last_peered":"2024-03-21T08:43:34.073626+0000","last_clean":"2024-03-21T08:43:34.073626+0000","last_became_active":"2024-03-21T08:43:34.068765+0000","last_became_peered":"2024-03-21T08:43:34.068765+0000","last_unstale":"2024-03-21T08:43:34.073626+0000","last_undegraded":"2024-03-21T08:43:34.073626+0000","last_fullsized":"2024-03-21T08:43:34.073626+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:28.749067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083560+0000","last_change":"2024-03-21T08:43:34.092372+0000","last_active":"2024-03-21T08:43:35.083560+0000","last_peered":"2024-03-21T08:43:35.083560+0000","last_clean":"2024-03-21T08:43:35.083560+0000","last_became_active":"2024-03-21T08:43:34.092101+0000","last_became_peered":"2024-03-21T08:43:34.092101+0000","last_unstale":"2024-03-21T08:43:35.083560+0000","last_undegraded":"2024-03-21T08:43:35.083560+0000","last_fullsized":"2024-03-21T08:43:35.083560+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:13:14.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084006+0000","last_change":"2024-03-21T08:43:34.087154+0000","last_active":"2024-03-21T08:43:35.084006+0000","last_peered":"2024-03-21T08:43:35.084006+0000","last_clean":"2024-03-21T08:43:35.084006+0000","last_became_active":"2024-03-21T08:43:34.086887+0000","last_became_peered":"2024-03-21T08:43:34.086887+0000","last_unstale":"2024-03-21T08:43:35.084006+0000","last_undegraded":"2024-03-21T08:43:35.084006+0000","last_fullsized":"2024-03-21T08:43:35.084006+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:33:07.228390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"53'4","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084217+0000","last_change":"2024-03-21T08:43:33.081655+0000","last_active":"2024-03-21T08:43:35.084217+0000","last_peered":"2024-03-21T08:43:35.084217+0000","last_clean":"2024-03-21T08:43:35.084217+0000","last_became_active":"2024-03-21T08:43:33.081260+0000","last_became_peered":"2024-03-21T08:43:33.081260+0000","last_unstale":"2024-03-21T08:43:35.084217+0000","last_undegraded":"2024-03-21T08:43:35.084217+0000","last_fullsized":"2024-03-21T08:43:35.084217+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.074406+0000","last_change":"2024-03-21T08:43:34.070772+0000","last_active":"2024-03-21T08:43:34.074406+0000","last_peered":"2024-03-21T08:43:34.074406+0000","last_clean":"2024-03-21T08:43:34.074406+0000","last_became_active":"2024-03-21T08:43:34.070484+0000","last_became_peered":"2024-03-21T08:43:34.070484+0000","last_unstale":"2024-03-21T08:43:34.074406+0000","last_undegraded":"2024-03-21T08:43:34.074406+0000","last_fullsized":"2024-03-21T08:43:34.074406+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:12:24.539063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267165+0000","last_change":"2024-03-21T08:43:33.078482+0000","last_active":"2024-03-21T08:43:35.267165+0000","last_peered":"2024-03-21T08:43:35.267165+0000","last_clean":"2024-03-21T08:43:35.267165+0000","last_became_active":"2024-03-21T08:43:33.078221+0000","last_became_peered":"2024-03-21T08:43:33.078221+0000","last_unstale":"2024-03-21T08:43:35.267165+0000","last_undegraded":"2024-03-21T08:43:35.267165+0000","last_fullsized":"2024-03-21T08:43:35.267165+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:52:15.693236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084200+0000","last_change":"2024-03-21T08:43:34.086549+0000","last_active":"2024-03-21T08:43:35.084200+0000","last_peered":"2024-03-21T08:43:35.084200+0000","last_clean":"2024-03-21T08:43:35.084200+0000","last_became_active":"2024-03-21T08:43:34.086138+0000","last_became_peered":"2024-03-21T08:43:34.086138+0000","last_unstale":"2024-03-21T08:43:35.084200+0000","last_undegraded":"2024-03-21T08:43:35.084200+0000","last_fullsized":"2024-03-21T08:43:35.084200+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:28:55.959297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084084+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:35.084084+0000","last_peered":"2024-03-21T08:43:35.084084+0000","last_clean":"2024-03-21T08:43:35.084084+0000","last_became_active":"2024-03-21T08:43:33.080225+0000","last_became_peered":"2024-03-21T08:43:33.080225+0000","last_unstale":"2024-03-21T08:43:35.084084+0000","last_undegraded":"2024-03-21T08:43:35.084084+0000","last_fullsized":"2024-03-21T08:43:35.084084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083978+0000","last_change":"2024-03-21T08:43:34.087074+0000","last_active":"2024-03-21T08:43:35.083978+0000","last_peered":"2024-03-21T08:43:35.083978+0000","last_clean":"2024-03-21T08:43:35.083978+0000","last_became_active":"2024-03-21T08:43:34.086751+0000","last_became_peered":"2024-03-21T08:43:34.086751+0000","last_unstale":"2024-03-21T08:43:35.083978+0000","last_undegraded":"2024-03-21T08:43:35.083978+0000","last_fullsized":"2024-03-21T08:43:35.083978+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065338+0000","last_change":"2024-03-21T08:43:33.083210+0000","last_active":"2024-03-21T08:43:35.065338+0000","last_peered":"2024-03-21T08:43:35.065338+0000","last_clean":"2024-03-21T08:43:35.065338+0000","last_became_active":"2024-03-21T08:43:33.082822+0000","last_became_peered":"2024-03-21T08:43:33.082822+0000","last_unstale":"2024-03-21T08:43:35.065338+0000","last_undegraded":"2024-03-21T08:43:35.065338+0000","last_fullsized":"2024-03-21T08:43:35.065338+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266331+0000","last_change":"2024-03-21T08:43:34.087514+0000","last_active":"2024-03-21T08:43:35.266331+0000","last_peered":"2024-03-21T08:43:35.266331+0000","last_clean":"2024-03-21T08:43:35.266331+0000","last_became_active":"2024-03-21T08:43:34.087212+0000","last_became_peered":"2024-03-21T08:43:34.087212+0000","last_unstale":"2024-03-21T08:43:35.266331+0000","last_undegraded":"2024-03-21T08:43:35.266331+0000","last_fullsized":"2024-03-21T08:43:35.266331+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:33:07.228390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265371+0000","last_change":"2024-03-21T08:43:33.078204+0000","last_active":"2024-03-21T08:43:35.265371+0000","last_peered":"2024-03-21T08:43:35.265371+0000","last_clean":"2024-03-21T08:43:35.265371+0000","last_became_active":"2024-03-21T08:43:33.077529+0000","last_became_peered":"2024-03-21T08:43:33.077529+0000","last_unstale":"2024-03-21T08:43:35.265371+0000","last_undegraded":"2024-03-21T08:43:35.265371+0000","last_fullsized":"2024-03-21T08:43:35.265371+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:11:04.693260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266690+0000","last_change":"2024-03-21T08:43:34.099085+0000","last_active":"2024-03-21T08:43:35.266690+0000","last_peered":"2024-03-21T08:43:35.266690+0000","last_clean":"2024-03-21T08:43:35.266690+0000","last_became_active":"2024-03-21T08:43:34.096614+0000","last_became_peered":"2024-03-21T08:43:34.096614+0000","last_unstale":"2024-03-21T08:43:35.266690+0000","last_undegraded":"2024-03-21T08:43:35.266690+0000","last_fullsized":"2024-03-21T08:43:35.266690+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:33.567206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083563+0000","last_change":"2024-03-21T08:43:33.078883+0000","last_active":"2024-03-21T08:43:35.083563+0000","last_peered":"2024-03-21T08:43:35.083563+0000","last_clean":"2024-03-21T08:43:35.083563+0000","last_became_active":"2024-03-21T08:43:33.078618+0000","last_became_peered":"2024-03-21T08:43:33.078618+0000","last_unstale":"2024-03-21T08:43:35.083563+0000","last_undegraded":"2024-03-21T08:43:35.083563+0000","last_fullsized":"2024-03-21T08:43:35.083563+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084441+0000","last_change":"2024-03-21T08:43:34.082546+0000","last_active":"2024-03-21T08:43:34.084441+0000","last_peered":"2024-03-21T08:43:34.084441+0000","last_clean":"2024-03-21T08:43:34.084441+0000","last_became_active":"2024-03-21T08:43:34.082250+0000","last_became_peered":"2024-03-21T08:43:34.082250+0000","last_unstale":"2024-03-21T08:43:34.084441+0000","last_undegraded":"2024-03-21T08:43:34.084441+0000","last_fullsized":"2024-03-21T08:43:34.084441+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:35.232127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064603+0000","last_change":"2024-03-21T08:43:33.075815+0000","last_active":"2024-03-21T08:43:35.064603+0000","last_peered":"2024-03-21T08:43:35.064603+0000","last_clean":"2024-03-21T08:43:35.064603+0000","last_became_active":"2024-03-21T08:43:33.075533+0000","last_became_peered":"2024-03-21T08:43:33.075533+0000","last_unstale":"2024-03-21T08:43:35.064603+0000","last_undegraded":"2024-03-21T08:43:35.064603+0000","last_fullsized":"2024-03-21T08:43:35.064603+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266477+0000","last_change":"2024-03-21T08:43:34.086730+0000","last_active":"2024-03-21T08:43:35.266477+0000","last_peered":"2024-03-21T08:43:35.266477+0000","last_clean":"2024-03-21T08:43:35.266477+0000","last_became_active":"2024-03-21T08:43:34.086471+0000","last_became_peered":"2024-03-21T08:43:34.086471+0000","last_unstale":"2024-03-21T08:43:35.266477+0000","last_undegraded":"2024-03-21T08:43:35.266477+0000","last_fullsized":"2024-03-21T08:43:35.266477+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:00:52.228139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265568+0000","last_change":"2024-03-21T08:43:33.076297+0000","last_active":"2024-03-21T08:43:35.265568+0000","last_peered":"2024-03-21T08:43:35.265568+0000","last_clean":"2024-03-21T08:43:35.265568+0000","last_became_active":"2024-03-21T08:43:33.075418+0000","last_became_peered":"2024-03-21T08:43:33.075418+0000","last_unstale":"2024-03-21T08:43:35.265568+0000","last_undegraded":"2024-03-21T08:43:35.265568+0000","last_fullsized":"2024-03-21T08:43:35.265568+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:20:51.363983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265878+0000","last_change":"2024-03-21T08:43:34.083842+0000","last_active":"2024-03-21T08:43:35.265878+0000","last_peered":"2024-03-21T08:43:35.265878+0000","last_clean":"2024-03-21T08:43:35.265878+0000","last_became_active":"2024-03-21T08:43:34.083499+0000","last_became_peered":"2024-03-21T08:43:34.083499+0000","last_unstale":"2024-03-21T08:43:35.265878+0000","last_undegraded":"2024-03-21T08:43:35.265878+0000","last_fullsized":"2024-03-21T08:43:35.265878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:42.706198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.066551+0000","last_change":"2024-03-21T08:43:33.084904+0000","last_active":"2024-03-21T08:43:34.066551+0000","last_peered":"2024-03-21T08:43:34.066551+0000","last_clean":"2024-03-21T08:43:34.066551+0000","last_became_active":"2024-03-21T08:43:33.084490+0000","last_became_peered":"2024-03-21T08:43:33.084490+0000","last_unstale":"2024-03-21T08:43:34.066551+0000","last_undegraded":"2024-03-21T08:43:34.066551+0000","last_fullsized":"2024-03-21T08:43:34.066551+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:59.934028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080558+0000","last_change":"2024-03-21T08:43:34.073629+0000","last_active":"2024-03-21T08:43:34.080558+0000","last_peered":"2024-03-21T08:43:34.080558+0000","last_clean":"2024-03-21T08:43:34.080558+0000","last_became_active":"2024-03-21T08:43:34.073264+0000","last_became_peered":"2024-03-21T08:43:34.073264+0000","last_unstale":"2024-03-21T08:43:34.080558+0000","last_undegraded":"2024-03-21T08:43:34.080558+0000","last_fullsized":"2024-03-21T08:43:34.080558+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:03:33.956736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267268+0000","last_change":"2024-03-21T08:43:33.080472+0000","last_active":"2024-03-21T08:43:35.267268+0000","last_peered":"2024-03-21T08:43:35.267268+0000","last_clean":"2024-03-21T08:43:35.267268+0000","last_became_active":"2024-03-21T08:43:33.080295+0000","last_became_peered":"2024-03-21T08:43:33.080295+0000","last_unstale":"2024-03-21T08:43:35.267268+0000","last_undegraded":"2024-03-21T08:43:35.267268+0000","last_fullsized":"2024-03-21T08:43:35.267268+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:03:50.003394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084312+0000","last_change":"2024-03-21T08:43:34.082659+0000","last_active":"2024-03-21T08:43:34.084312+0000","last_peered":"2024-03-21T08:43:34.084312+0000","last_clean":"2024-03-21T08:43:34.084312+0000","last_became_active":"2024-03-21T08:43:34.082297+0000","last_became_peered":"2024-03-21T08:43:34.082297+0000","last_unstale":"2024-03-21T08:43:34.084312+0000","last_undegraded":"2024-03-21T08:43:34.084312+0000","last_fullsized":"2024-03-21T08:43:34.084312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:58:25.147246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"53'3","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.100215+0000","last_change":"2024-03-21T08:43:33.079062+0000","last_active":"2024-03-21T08:43:34.100215+0000","last_peered":"2024-03-21T08:43:34.100215+0000","last_clean":"2024-03-21T08:43:34.100215+0000","last_became_active":"2024-03-21T08:43:33.078759+0000","last_became_peered":"2024-03-21T08:43:33.078759+0000","last_unstale":"2024-03-21T08:43:34.100215+0000","last_undegraded":"2024-03-21T08:43:34.100215+0000","last_fullsized":"2024-03-21T08:43:34.100215+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:09.919722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.079916+0000","last_change":"2024-03-21T08:43:34.074492+0000","last_active":"2024-03-21T08:43:34.079916+0000","last_peered":"2024-03-21T08:43:34.079916+0000","last_clean":"2024-03-21T08:43:34.079916+0000","last_became_active":"2024-03-21T08:43:34.074094+0000","last_became_peered":"2024-03-21T08:43:34.074094+0000","last_unstale":"2024-03-21T08:43:34.079916+0000","last_undegraded":"2024-03-21T08:43:34.079916+0000","last_fullsized":"2024-03-21T08:43:34.079916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:53:30.468809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"53'6","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265735+0000","last_change":"2024-03-21T08:43:33.074842+0000","last_active":"2024-03-21T08:43:35.265735+0000","last_peered":"2024-03-21T08:43:35.265735+0000","last_clean":"2024-03-21T08:43:35.265735+0000","last_became_active":"2024-03-21T08:43:33.074382+0000","last_became_peered":"2024-03-21T08:43:33.074382+0000","last_unstale":"2024-03-21T08:43:35.265735+0000","last_undegraded":"2024-03-21T08:43:35.265735+0000","last_fullsized":"2024-03-21T08:43:35.265735+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:17.408018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064825+0000","last_change":"2024-03-21T08:43:34.070945+0000","last_active":"2024-03-21T08:43:35.064825+0000","last_peered":"2024-03-21T08:43:35.064825+0000","last_clean":"2024-03-21T08:43:35.064825+0000","last_became_active":"2024-03-21T08:43:34.070613+0000","last_became_peered":"2024-03-21T08:43:34.070613+0000","last_unstale":"2024-03-21T08:43:35.064825+0000","last_undegraded":"2024-03-21T08:43:35.064825+0000","last_fullsized":"2024-03-21T08:43:35.064825+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:31:37.489435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266714+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:35.266714+0000","last_peered":"2024-03-21T08:43:35.266714+0000","last_clean":"2024-03-21T08:43:35.266714+0000","last_became_active":"2024-03-21T08:43:33.080224+0000","last_became_peered":"2024-03-21T08:43:33.080224+0000","last_unstale":"2024-03-21T08:43:35.266714+0000","last_undegraded":"2024-03-21T08:43:35.266714+0000","last_fullsized":"2024-03-21T08:43:35.266714+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.080983+0000","last_change":"2024-03-21T08:43:34.078879+0000","last_active":"2024-03-21T08:43:34.080983+0000","last_peered":"2024-03-21T08:43:34.080983+0000","last_clean":"2024-03-21T08:43:34.080983+0000","last_became_active":"2024-03-21T08:43:34.078610+0000","last_became_peered":"2024-03-21T08:43:34.078610+0000","last_unstale":"2024-03-21T08:43:34.080983+0000","last_undegraded":"2024-03-21T08:43:34.080983+0000","last_fullsized":"2024-03-21T08:43:34.080983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:57:32.786072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072634+0000","last_change":"2024-03-21T08:43:33.085060+0000","last_active":"2024-03-21T08:43:34.072634+0000","last_peered":"2024-03-21T08:43:34.072634+0000","last_clean":"2024-03-21T08:43:34.072634+0000","last_became_active":"2024-03-21T08:43:33.084725+0000","last_became_peered":"2024-03-21T08:43:33.084725+0000","last_unstale":"2024-03-21T08:43:34.072634+0000","last_undegraded":"2024-03-21T08:43:34.072634+0000","last_fullsized":"2024-03-21T08:43:34.072634+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265565+0000","last_change":"2024-03-21T08:43:34.085498+0000","last_active":"2024-03-21T08:43:35.265565+0000","last_peered":"2024-03-21T08:43:35.265565+0000","last_clean":"2024-03-21T08:43:35.265565+0000","last_became_active":"2024-03-21T08:43:34.085185+0000","last_became_peered":"2024-03-21T08:43:34.085185+0000","last_unstale":"2024-03-21T08:43:35.265565+0000","last_undegraded":"2024-03-21T08:43:35.265565+0000","last_fullsized":"2024-03-21T08:43:35.265565+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:41.809827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064697+0000","last_change":"2024-03-21T08:43:33.075978+0000","last_active":"2024-03-21T08:43:35.064697+0000","last_peered":"2024-03-21T08:43:35.064697+0000","last_clean":"2024-03-21T08:43:35.064697+0000","last_became_active":"2024-03-21T08:43:33.075230+0000","last_became_peered":"2024-03-21T08:43:33.075230+0000","last_unstale":"2024-03-21T08:43:35.064697+0000","last_undegraded":"2024-03-21T08:43:35.064697+0000","last_fullsized":"2024-03-21T08:43:35.064697+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.088395+0000","last_change":"2024-03-21T08:43:34.081971+0000","last_active":"2024-03-21T08:43:34.088395+0000","last_peered":"2024-03-21T08:43:34.088395+0000","last_clean":"2024-03-21T08:43:34.088395+0000","last_became_active":"2024-03-21T08:43:34.080960+0000","last_became_peered":"2024-03-21T08:43:34.080960+0000","last_unstale":"2024-03-21T08:43:34.088395+0000","last_undegraded":"2024-03-21T08:43:34.088395+0000","last_fullsized":"2024-03-21T08:43:34.088395+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:37:54.133660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"53'8","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.098747+0000","last_change":"2024-03-21T08:43:33.082199+0000","last_active":"2024-03-21T08:43:34.098747+0000","last_peered":"2024-03-21T08:43:34.098747+0000","last_clean":"2024-03-21T08:43:34.098747+0000","last_became_active":"2024-03-21T08:43:33.081892+0000","last_became_peered":"2024-03-21T08:43:33.081892+0000","last_unstale":"2024-03-21T08:43:34.098747+0000","last_undegraded":"2024-03-21T08:43:34.098747+0000","last_fullsized":"2024-03-21T08:43:34.098747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:19.722045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.092883+0000","last_change":"2024-03-21T08:43:34.086279+0000","last_active":"2024-03-21T08:43:34.092883+0000","last_peered":"2024-03-21T08:43:34.092883+0000","last_clean":"2024-03-21T08:43:34.092883+0000","last_became_active":"2024-03-21T08:43:34.086021+0000","last_became_peered":"2024-03-21T08:43:34.086021+0000","last_unstale":"2024-03-21T08:43:34.092883+0000","last_undegraded":"2024-03-21T08:43:34.092883+0000","last_fullsized":"2024-03-21T08:43:34.092883+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:04:51.667945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267343+0000","last_change":"2024-03-21T08:43:33.082173+0000","last_active":"2024-03-21T08:43:35.267343+0000","last_peered":"2024-03-21T08:43:35.267343+0000","last_clean":"2024-03-21T08:43:35.267343+0000","last_became_active":"2024-03-21T08:43:33.081902+0000","last_became_peered":"2024-03-21T08:43:33.081902+0000","last_unstale":"2024-03-21T08:43:35.267343+0000","last_undegraded":"2024-03-21T08:43:35.267343+0000","last_fullsized":"2024-03-21T08:43:35.267343+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:29:40.400581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065436+0000","last_change":"2024-03-21T08:43:34.070653+0000","last_active":"2024-03-21T08:43:35.065436+0000","last_peered":"2024-03-21T08:43:35.065436+0000","last_clean":"2024-03-21T08:43:35.065436+0000","last_became_active":"2024-03-21T08:43:34.070064+0000","last_became_peered":"2024-03-21T08:43:34.070064+0000","last_unstale":"2024-03-21T08:43:35.065436+0000","last_undegraded":"2024-03-21T08:43:35.065436+0000","last_fullsized":"2024-03-21T08:43:35.065436+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:03.252966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084043+0000","last_change":"2024-03-21T08:43:33.077524+0000","last_active":"2024-03-21T08:43:35.084043+0000","last_peered":"2024-03-21T08:43:35.084043+0000","last_clean":"2024-03-21T08:43:35.084043+0000","last_became_active":"2024-03-21T08:43:33.077163+0000","last_became_peered":"2024-03-21T08:43:33.077163+0000","last_unstale":"2024-03-21T08:43:35.084043+0000","last_undegraded":"2024-03-21T08:43:35.084043+0000","last_fullsized":"2024-03-21T08:43:35.084043+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267185+0000","last_change":"2024-03-21T08:43:34.088073+0000","last_active":"2024-03-21T08:43:35.267185+0000","last_peered":"2024-03-21T08:43:35.267185+0000","last_clean":"2024-03-21T08:43:35.267185+0000","last_became_active":"2024-03-21T08:43:34.087799+0000","last_became_peered":"2024-03-21T08:43:34.087799+0000","last_unstale":"2024-03-21T08:43:35.267185+0000","last_undegraded":"2024-03-21T08:43:35.267185+0000","last_fullsized":"2024-03-21T08:43:35.267185+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:13.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065165+0000","last_change":"2024-03-21T08:43:33.078372+0000","last_active":"2024-03-21T08:43:35.065165+0000","last_peered":"2024-03-21T08:43:35.065165+0000","last_clean":"2024-03-21T08:43:35.065165+0000","last_became_active":"2024-03-21T08:43:33.077797+0000","last_became_peered":"2024-03-21T08:43:33.077797+0000","last_unstale":"2024-03-21T08:43:35.065165+0000","last_undegraded":"2024-03-21T08:43:35.065165+0000","last_fullsized":"2024-03-21T08:43:35.065165+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.076176+0000","last_change":"2024-03-21T08:43:34.072036+0000","last_active":"2024-03-21T08:43:34.076176+0000","last_peered":"2024-03-21T08:43:34.076176+0000","last_clean":"2024-03-21T08:43:34.076176+0000","last_became_active":"2024-03-21T08:43:34.071710+0000","last_became_peered":"2024-03-21T08:43:34.071710+0000","last_unstale":"2024-03-21T08:43:34.076176+0000","last_undegraded":"2024-03-21T08:43:34.076176+0000","last_fullsized":"2024-03-21T08:43:34.076176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:37.084279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083881+0000","last_change":"2024-03-21T08:43:33.079120+0000","last_active":"2024-03-21T08:43:35.083881+0000","last_peered":"2024-03-21T08:43:35.083881+0000","last_clean":"2024-03-21T08:43:35.083881+0000","last_became_active":"2024-03-21T08:43:33.078672+0000","last_became_peered":"2024-03-21T08:43:33.078672+0000","last_unstale":"2024-03-21T08:43:35.083881+0000","last_undegraded":"2024-03-21T08:43:35.083881+0000","last_fullsized":"2024-03-21T08:43:35.083881+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064890+0000","last_change":"2024-03-21T08:43:34.073220+0000","last_active":"2024-03-21T08:43:35.064890+0000","last_peered":"2024-03-21T08:43:35.064890+0000","last_clean":"2024-03-21T08:43:35.064890+0000","last_became_active":"2024-03-21T08:43:34.072909+0000","last_became_peered":"2024-03-21T08:43:34.072909+0000","last_unstale":"2024-03-21T08:43:35.064890+0000","last_undegraded":"2024-03-21T08:43:35.064890+0000","last_fullsized":"2024-03-21T08:43:35.064890+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:25.666989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266922+0000","last_change":"2024-03-21T08:43:33.074311+0000","last_active":"2024-03-21T08:43:35.266922+0000","last_peered":"2024-03-21T08:43:35.266922+0000","last_clean":"2024-03-21T08:43:35.266922+0000","last_became_active":"2024-03-21T08:43:33.074047+0000","last_became_peered":"2024-03-21T08:43:33.074047+0000","last_unstale":"2024-03-21T08:43:35.266922+0000","last_undegraded":"2024-03-21T08:43:35.266922+0000","last_fullsized":"2024-03-21T08:43:35.266922+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083835+0000","last_change":"2024-03-21T08:43:34.090009+0000","last_active":"2024-03-21T08:43:35.083835+0000","last_peered":"2024-03-21T08:43:35.083835+0000","last_clean":"2024-03-21T08:43:35.083835+0000","last_became_active":"2024-03-21T08:43:34.089356+0000","last_became_peered":"2024-03-21T08:43:34.089356+0000","last_unstale":"2024-03-21T08:43:35.083835+0000","last_undegraded":"2024-03-21T08:43:35.083835+0000","last_fullsized":"2024-03-21T08:43:35.083835+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:20:04.245491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.062005+0000","last_change":"2024-03-21T08:43:33.080008+0000","last_active":"2024-03-21T08:43:34.062005+0000","last_peered":"2024-03-21T08:43:34.062005+0000","last_clean":"2024-03-21T08:43:34.062005+0000","last_became_active":"2024-03-21T08:43:33.079744+0000","last_became_peered":"2024-03-21T08:43:33.079744+0000","last_unstale":"2024-03-21T08:43:34.062005+0000","last_undegraded":"2024-03-21T08:43:34.062005+0000","last_fullsized":"2024-03-21T08:43:34.062005+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:34.225900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083515+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:35.083515+0000","last_peered":"2024-03-21T08:43:35.083515+0000","last_clean":"2024-03-21T08:43:35.083515+0000","last_became_active":"2024-03-21T08:43:34.089398+0000","last_became_peered":"2024-03-21T08:43:34.089398+0000","last_unstale":"2024-03-21T08:43:35.083515+0000","last_undegraded":"2024-03-21T08:43:35.083515+0000","last_fullsized":"2024-03-21T08:43:35.083515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:20:04.099452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064150+0000","last_change":"2024-03-21T08:43:33.076251+0000","last_active":"2024-03-21T08:43:35.064150+0000","last_peered":"2024-03-21T08:43:35.064150+0000","last_clean":"2024-03-21T08:43:35.064150+0000","last_became_active":"2024-03-21T08:43:33.074661+0000","last_became_peered":"2024-03-21T08:43:33.074661+0000","last_unstale":"2024-03-21T08:43:35.064150+0000","last_undegraded":"2024-03-21T08:43:35.064150+0000","last_fullsized":"2024-03-21T08:43:35.064150+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065344+0000","last_change":"2024-03-21T08:43:34.068766+0000","last_active":"2024-03-21T08:43:35.065344+0000","last_peered":"2024-03-21T08:43:35.065344+0000","last_clean":"2024-03-21T08:43:35.065344+0000","last_became_active":"2024-03-21T08:43:34.068447+0000","last_became_peered":"2024-03-21T08:43:34.068447+0000","last_unstale":"2024-03-21T08:43:35.065344+0000","last_undegraded":"2024-03-21T08:43:35.065344+0000","last_fullsized":"2024-03-21T08:43:35.065344+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:52.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"53'2","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.106185+0000","last_change":"2024-03-21T08:43:33.079309+0000","last_active":"2024-03-21T08:43:34.106185+0000","last_peered":"2024-03-21T08:43:34.106185+0000","last_clean":"2024-03-21T08:43:34.106185+0000","last_became_active":"2024-03-21T08:43:33.079038+0000","last_became_peered":"2024-03-21T08:43:33.079038+0000","last_unstale":"2024-03-21T08:43:34.106185+0000","last_undegraded":"2024-03-21T08:43:34.106185+0000","last_fullsized":"2024-03-21T08:43:34.106185+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:23.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065497+0000","last_change":"2024-03-21T08:43:34.079196+0000","last_active":"2024-03-21T08:43:35.065497+0000","last_peered":"2024-03-21T08:43:35.065497+0000","last_clean":"2024-03-21T08:43:35.065497+0000","last_became_active":"2024-03-21T08:43:34.078900+0000","last_became_peered":"2024-03-21T08:43:34.078900+0000","last_unstale":"2024-03-21T08:43:35.065497+0000","last_undegraded":"2024-03-21T08:43:35.065497+0000","last_fullsized":"2024-03-21T08:43:35.065497+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"54'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084143+0000","last_change":"2024-03-21T08:43:33.078636+0000","last_active":"2024-03-21T08:43:35.084143+0000","last_peered":"2024-03-21T08:43:35.084143+0000","last_clean":"2024-03-21T08:43:35.084143+0000","last_became_active":"2024-03-21T08:43:33.078336+0000","last_became_peered":"2024-03-21T08:43:33.078336+0000","last_unstale":"2024-03-21T08:43:35.084143+0000","last_undegraded":"2024-03-21T08:43:35.084143+0000","last_fullsized":"2024-03-21T08:43:35.084143+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.084289+0000","last_change":"2024-03-21T08:43:34.082469+0000","last_active":"2024-03-21T08:43:34.084289+0000","last_peered":"2024-03-21T08:43:34.084289+0000","last_clean":"2024-03-21T08:43:34.084289+0000","last_became_active":"2024-03-21T08:43:34.082227+0000","last_became_peered":"2024-03-21T08:43:34.082227+0000","last_unstale":"2024-03-21T08:43:34.084289+0000","last_undegraded":"2024-03-21T08:43:34.084289+0000","last_fullsized":"2024-03-21T08:43:34.084289+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:14.199100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.071019+0000","last_change":"2024-03-21T08:43:33.077725+0000","last_active":"2024-03-21T08:43:34.071019+0000","last_peered":"2024-03-21T08:43:34.071019+0000","last_clean":"2024-03-21T08:43:34.071019+0000","last_became_active":"2024-03-21T08:43:33.077452+0000","last_became_peered":"2024-03-21T08:43:33.077452+0000","last_unstale":"2024-03-21T08:43:34.071019+0000","last_undegraded":"2024-03-21T08:43:34.071019+0000","last_fullsized":"2024-03-21T08:43:34.071019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:22:33.956371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084098+0000","last_change":"2024-03-21T08:43:34.082367+0000","last_active":"2024-03-21T08:43:35.084098+0000","last_peered":"2024-03-21T08:43:35.084098+0000","last_clean":"2024-03-21T08:43:35.084098+0000","last_became_active":"2024-03-21T08:43:34.081770+0000","last_became_peered":"2024-03-21T08:43:34.081770+0000","last_unstale":"2024-03-21T08:43:35.084098+0000","last_undegraded":"2024-03-21T08:43:35.084098+0000","last_fullsized":"2024-03-21T08:43:35.084098+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:31:03.090585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065464+0000","last_change":"2024-03-21T08:43:33.076652+0000","last_active":"2024-03-21T08:43:35.065464+0000","last_peered":"2024-03-21T08:43:35.065464+0000","last_clean":"2024-03-21T08:43:35.065464+0000","last_became_active":"2024-03-21T08:43:33.074101+0000","last_became_peered":"2024-03-21T08:43:33.074101+0000","last_unstale":"2024-03-21T08:43:35.065464+0000","last_undegraded":"2024-03-21T08:43:35.065464+0000","last_fullsized":"2024-03-21T08:43:35.065464+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266463+0000","last_change":"2024-03-21T08:43:34.087659+0000","last_active":"2024-03-21T08:43:35.266463+0000","last_peered":"2024-03-21T08:43:35.266463+0000","last_clean":"2024-03-21T08:43:35.266463+0000","last_became_active":"2024-03-21T08:43:34.087310+0000","last_became_peered":"2024-03-21T08:43:34.087310+0000","last_unstale":"2024-03-21T08:43:35.266463+0000","last_undegraded":"2024-03-21T08:43:35.266463+0000","last_fullsized":"2024-03-21T08:43:35.266463+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:49:15.990964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061812+0000","last_change":"2024-03-21T08:43:33.081506+0000","last_active":"2024-03-21T08:43:34.061812+0000","last_peered":"2024-03-21T08:43:34.061812+0000","last_clean":"2024-03-21T08:43:34.061812+0000","last_became_active":"2024-03-21T08:43:33.080894+0000","last_became_peered":"2024-03-21T08:43:33.080894+0000","last_unstale":"2024-03-21T08:43:34.061812+0000","last_undegraded":"2024-03-21T08:43:34.061812+0000","last_fullsized":"2024-03-21T08:43:34.061812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.093878+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:34.093878+0000","last_peered":"2024-03-21T08:43:34.093878+0000","last_clean":"2024-03-21T08:43:34.093878+0000","last_became_active":"2024-03-21T08:43:34.089436+0000","last_became_peered":"2024-03-21T08:43:34.089436+0000","last_unstale":"2024-03-21T08:43:34.093878+0000","last_undegraded":"2024-03-21T08:43:34.093878+0000","last_fullsized":"2024-03-21T08:43:34.093878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:35:10.546830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265555+0000","last_change":"2024-03-21T08:43:33.080132+0000","last_active":"2024-03-21T08:43:35.265555+0000","last_peered":"2024-03-21T08:43:35.265555+0000","last_clean":"2024-03-21T08:43:35.265555+0000","last_became_active":"2024-03-21T08:43:33.079867+0000","last_became_peered":"2024-03-21T08:43:33.079867+0000","last_unstale":"2024-03-21T08:43:35.265555+0000","last_undegraded":"2024-03-21T08:43:35.265555+0000","last_fullsized":"2024-03-21T08:43:35.265555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:14:22.990820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.089682+0000","last_change":"2024-03-21T08:43:34.084715+0000","last_active":"2024-03-21T08:43:34.089682+0000","last_peered":"2024-03-21T08:43:34.089682+0000","last_clean":"2024-03-21T08:43:34.089682+0000","last_became_active":"2024-03-21T08:43:34.084436+0000","last_became_peered":"2024-03-21T08:43:34.084436+0000","last_unstale":"2024-03-21T08:43:34.089682+0000","last_undegraded":"2024-03-21T08:43:34.089682+0000","last_fullsized":"2024-03-21T08:43:34.089682+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:33:01.770928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.061783+0000","last_change":"2024-03-21T08:43:33.076471+0000","last_active":"2024-03-21T08:43:34.061783+0000","last_peered":"2024-03-21T08:43:34.061783+0000","last_clean":"2024-03-21T08:43:34.061783+0000","last_became_active":"2024-03-21T08:43:33.076185+0000","last_became_peered":"2024-03-21T08:43:33.076185+0000","last_unstale":"2024-03-21T08:43:34.061783+0000","last_undegraded":"2024-03-21T08:43:34.061783+0000","last_fullsized":"2024-03-21T08:43:34.061783+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266350+0000","last_change":"2024-03-21T08:43:34.087249+0000","last_active":"2024-03-21T08:43:35.266350+0000","last_peered":"2024-03-21T08:43:35.266350+0000","last_clean":"2024-03-21T08:43:35.266350+0000","last_became_active":"2024-03-21T08:43:34.085986+0000","last_became_peered":"2024-03-21T08:43:34.085986+0000","last_unstale":"2024-03-21T08:43:35.266350+0000","last_undegraded":"2024-03-21T08:43:35.266350+0000","last_fullsized":"2024-03-21T08:43:35.266350+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:31:03.090585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266389+0000","last_change":"2024-03-21T08:43:33.074819+0000","last_active":"2024-03-21T08:43:35.266389+0000","last_peered":"2024-03-21T08:43:35.266389+0000","last_clean":"2024-03-21T08:43:35.266389+0000","last_became_active":"2024-03-21T08:43:33.074585+0000","last_became_peered":"2024-03-21T08:43:33.074585+0000","last_unstale":"2024-03-21T08:43:35.266389+0000","last_undegraded":"2024-03-21T08:43:35.266389+0000","last_fullsized":"2024-03-21T08:43:35.266389+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266282+0000","last_change":"2024-03-21T08:43:34.099486+0000","last_active":"2024-03-21T08:43:35.266282+0000","last_peered":"2024-03-21T08:43:35.266282+0000","last_clean":"2024-03-21T08:43:35.266282+0000","last_became_active":"2024-03-21T08:43:34.096768+0000","last_became_peered":"2024-03-21T08:43:34.096768+0000","last_unstale":"2024-03-21T08:43:35.266282+0000","last_undegraded":"2024-03-21T08:43:35.266282+0000","last_fullsized":"2024-03-21T08:43:35.266282+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.072378+0000","last_change":"2024-03-21T08:43:33.081102+0000","last_active":"2024-03-21T08:43:34.072378+0000","last_peered":"2024-03-21T08:43:34.072378+0000","last_clean":"2024-03-21T08:43:34.072378+0000","last_became_active":"2024-03-21T08:43:33.080820+0000","last_became_peered":"2024-03-21T08:43:33.080820+0000","last_unstale":"2024-03-21T08:43:34.072378+0000","last_undegraded":"2024-03-21T08:43:34.072378+0000","last_fullsized":"2024-03-21T08:43:34.072378+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-21T08:43:34.065013+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:43:34.065013+0000","last_peered":"2024-03-21T08:43:34.065013+0000","last_clean":"2024-03-21T08:43:34.065013+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:43:34.065013+0000","last_undegraded":"2024-03-21T08:43:34.065013+0000","last_fullsized":"2024-03-21T08:43:34.065013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509476,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1228800,"data_stored":871737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":274042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":281209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":281299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":288553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542180,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":289513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542179,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1249280,"data_stored":880463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:43:42.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:43:43.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 Processor -- start 2024-03-21T08:43:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 -- start start 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd090099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0900cdb50 con 0x7fd0900c5f80 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0900cde30 con 0x7fd09015f260 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.066+0000 7fd095000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09009a520 con 0x7fd090099b80 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd08e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd0900c5f80 0x7fd0900c63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36826/0 (socket says 172.21.15.28:36826) 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd08e800640 1 -- 172.21.15.28:0/3010075429 learned_addr learned my addr 172.21.15.28:0/3010075429 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd085400640 1 --2- 172.21.15.28:0/3010075429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd090099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932172310 0 0) 0x7fd0900cdb50 con 0x7fd0900c5f80 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd074003530 con 0x7fd0900c5f80 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064947506 0 0) 0x7fd0900cde30 con 0x7fd09015f260 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd085400640 1 -- 172.21.15.28:0/3010075429 >> v1:172.21.15.151:6789/0 conn(0x7fd09015f260 legacy=0x7fd0900cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd085400640 1 -- 172.21.15.28:0/3010075429 >> v1:172.21.15.28:6789/0 conn(0x7fd0900c5f80 legacy=0x7fd0900c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.070+0000 7fd085400640 1 -- 172.21.15.28:0/3010075429 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900c5780 con 0x7fd090099b80 2024-03-21T08:43:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd085400640 1 --2- 172.21.15.28:0/3010075429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd090099fe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097f0 tx=0x7fd07801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22806c68b0997665 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2070 con 0x7fd090099b80 2024-03-21T08:43:43.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0780a9070 con 0x7fd090099b80 2024-03-21T08:43:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd08d400640 1 -- 172.21.15.28:0/3010075429 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780ad880 con 0x7fd090099b80 2024-03-21T08:43:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 -- 172.21.15.28:0/3010075429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 msgr2=0x7fd090099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 --2- 172.21.15.28:0/3010075429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd090099fe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd0780097f0 tx=0x7fd07801c450 comp rx=0 tx=0).stop 2024-03-21T08:43:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 -- 172.21.15.28:0/3010075429 shutdown_connections 2024-03-21T08:43:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 --2- 172.21.15.28:0/3010075429 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd090099fe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 -- 172.21.15.28:0/3010075429 >> 172.21.15.28:0/3010075429 conn(0x7fd0900b34d0 msgr2=0x7fd0900b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 -- 172.21.15.28:0/3010075429 shutdown_connections 2024-03-21T08:43:43.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.074+0000 7fd095000640 1 -- 172.21.15.28:0/3010075429 wait complete. 2024-03-21T08:43:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 Processor -- start 2024-03-21T08:43:43.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- start start 2024-03-21T08:43:43.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd0900cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 0x7fd09014dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09015f7c0 con 0x7fd0900c5f80 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09015f640 con 0x7fd09015f260 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd090169a50 con 0x7fd090099b80 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50580/0 (socket says 172.21.15.28:50580) 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 -- 172.21.15.28:0/202039344 learned_addr learned my addr 172.21.15.28:0/202039344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd08de00640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd0900cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd08e800640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 0x7fd09014dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 -- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 msgr2=0x7fd0900cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd0900cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 -- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 msgr2=0x7fd09014dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 0x7fd09014dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 -- 172.21.15.28:0/202039344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd090150960 con 0x7fd0900c5f80 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd08e800640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 0x7fd09014dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd085400640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd0780b0040 tx=0x7fd07801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b2050 con 0x7fd0900c5f80 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0780b3040 con 0x7fd0900c5f80 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b9450 con 0x7fd0900c5f80 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd090150c40 con 0x7fd0900c5f80 2024-03-21T08:43:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd090151200 con 0x7fd0900c5f80 2024-03-21T08:43:43.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.078+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09004f410 con 0x7fd0900c5f80 2024-03-21T08:43:43.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.082+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd0780c0030 con 0x7fd0900c5f80 2024-03-21T08:43:43.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.082+0000 7fd086a00640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 0x7fd06c06a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.082+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd07811fa20 con 0x7fd0900c5f80 2024-03-21T08:43:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.082+0000 7fd08de00640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 0x7fd06c06a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.082+0000 7fd08de00640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 0x7fd06c06a0b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd090146740 tx=0x7fd080009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.090+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd0780160d0 con 0x7fd0900c5f80 2024-03-21T08:43:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.377+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0900ccc30 con 0x7fd06c067bf0 2024-03-21T08:43:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.385+0000 7fd086a00640 1 -- 172.21.15.28:0/202039344 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251943 (secure 0 0 0) 0x7fd0900ccc30 con 0x7fd06c067bf0 2024-03-21T08:43:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:43:43.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 msgr2=0x7fd06c06a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 0x7fd06c06a0b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd090146740 tx=0x7fd080009040 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 msgr2=0x7fd090151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd0780b0040 tx=0x7fd07801ac90 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 shutdown_connections 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd09015f260 0x7fd09014dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0900c5f80 0x7fd090151a00 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd06c067bf0 0x7fd06c06a0b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 --2- 172.21.15.28:0/202039344 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd090099b80 0x7fd0900cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 >> 172.21.15.28:0/202039344 conn(0x7fd0900b34d0 msgr2=0x7fd09015cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 shutdown_connections 2024-03-21T08:43:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.389+0000 7fd095000640 1 -- 172.21.15.28:0/202039344 wait complete. 2024-03-21T08:43:43.412 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-21T08:43:41.754617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239056,"kb_used_data":6216,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735664,"statfs":{"total":773094113280,"available":772849319936,"internally_reserved":0,"allocated":6365184,"data_stored":3488764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001993"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267368+0000","last_change":"2024-03-21T08:43:34.104498+0000","last_active":"2024-03-21T08:43:35.267368+0000","last_peered":"2024-03-21T08:43:35.267368+0000","last_clean":"2024-03-21T08:43:35.267368+0000","last_became_active":"2024-03-21T08:43:34.102336+0000","last_became_peered":"2024-03-21T08:43:34.102336+0000","last_unstale":"2024-03-21T08:43:35.267368+0000","last_undegraded":"2024-03-21T08:43:35.267368+0000","last_fullsized":"2024-03-21T08:43:35.267368+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:48:56.136586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267179+0000","last_change":"2024-03-21T08:43:34.088311+0000","last_active":"2024-03-21T08:43:35.267179+0000","last_peered":"2024-03-21T08:43:35.267179+0000","last_clean":"2024-03-21T08:43:35.267179+0000","last_became_active":"2024-03-21T08:43:34.087919+0000","last_became_peered":"2024-03-21T08:43:34.087919+0000","last_unstale":"2024-03-21T08:43:35.267179+0000","last_undegraded":"2024-03-21T08:43:35.267179+0000","last_fullsized":"2024-03-21T08:43:35.267179+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:32:01.332600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064489+0000","last_change":"2024-03-21T08:43:34.079798+0000","last_active":"2024-03-21T08:43:35.064489+0000","last_peered":"2024-03-21T08:43:35.064489+0000","last_clean":"2024-03-21T08:43:35.064489+0000","last_became_active":"2024-03-21T08:43:34.079557+0000","last_became_peered":"2024-03-21T08:43:34.079557+0000","last_unstale":"2024-03-21T08:43:35.064489+0000","last_undegraded":"2024-03-21T08:43:35.064489+0000","last_fullsized":"2024-03-21T08:43:35.064489+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:01:22.561928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065349+0000","last_change":"2024-03-21T08:43:34.069789+0000","last_active":"2024-03-21T08:43:35.065349+0000","last_peered":"2024-03-21T08:43:35.065349+0000","last_clean":"2024-03-21T08:43:35.065349+0000","last_became_active":"2024-03-21T08:43:34.068694+0000","last_became_peered":"2024-03-21T08:43:34.068694+0000","last_unstale":"2024-03-21T08:43:35.065349+0000","last_undegraded":"2024-03-21T08:43:35.065349+0000","last_fullsized":"2024-03-21T08:43:35.065349+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:20:04.245491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266588+0000","last_change":"2024-03-21T08:43:34.069862+0000","last_active":"2024-03-21T08:43:35.266588+0000","last_peered":"2024-03-21T08:43:35.266588+0000","last_clean":"2024-03-21T08:43:35.266588+0000","last_became_active":"2024-03-21T08:43:34.069561+0000","last_became_peered":"2024-03-21T08:43:34.069561+0000","last_unstale":"2024-03-21T08:43:35.266588+0000","last_undegraded":"2024-03-21T08:43:35.266588+0000","last_fullsized":"2024-03-21T08:43:35.266588+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:59:34.238327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083581+0000","last_change":"2024-03-21T08:43:34.104956+0000","last_active":"2024-03-21T08:43:35.083581+0000","last_peered":"2024-03-21T08:43:35.083581+0000","last_clean":"2024-03-21T08:43:35.083581+0000","last_became_active":"2024-03-21T08:43:34.104667+0000","last_became_peered":"2024-03-21T08:43:34.104667+0000","last_unstale":"2024-03-21T08:43:35.083581+0000","last_undegraded":"2024-03-21T08:43:35.083581+0000","last_fullsized":"2024-03-21T08:43:35.083581+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:52.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265410+0000","last_change":"2024-03-21T08:43:34.086151+0000","last_active":"2024-03-21T08:43:35.265410+0000","last_peered":"2024-03-21T08:43:35.265410+0000","last_clean":"2024-03-21T08:43:35.265410+0000","last_became_active":"2024-03-21T08:43:34.085809+0000","last_became_peered":"2024-03-21T08:43:34.085809+0000","last_unstale":"2024-03-21T08:43:35.265410+0000","last_undegraded":"2024-03-21T08:43:35.265410+0000","last_fullsized":"2024-03-21T08:43:35.265410+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:03:40.388982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266351+0000","last_change":"2024-03-21T08:43:34.083980+0000","last_active":"2024-03-21T08:43:35.266351+0000","last_peered":"2024-03-21T08:43:35.266351+0000","last_clean":"2024-03-21T08:43:35.266351+0000","last_became_active":"2024-03-21T08:43:34.083638+0000","last_became_peered":"2024-03-21T08:43:34.083638+0000","last_unstale":"2024-03-21T08:43:35.266351+0000","last_undegraded":"2024-03-21T08:43:35.266351+0000","last_fullsized":"2024-03-21T08:43:35.266351+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:28:55.959297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265818+0000","last_change":"2024-03-21T08:43:34.090630+0000","last_active":"2024-03-21T08:43:35.265818+0000","last_peered":"2024-03-21T08:43:35.265818+0000","last_clean":"2024-03-21T08:43:35.265818+0000","last_became_active":"2024-03-21T08:43:34.090323+0000","last_became_peered":"2024-03-21T08:43:34.090323+0000","last_unstale":"2024-03-21T08:43:35.265818+0000","last_undegraded":"2024-03-21T08:43:35.265818+0000","last_fullsized":"2024-03-21T08:43:35.265818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:13:22.184827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063858+0000","last_change":"2024-03-21T08:43:34.076812+0000","last_active":"2024-03-21T08:43:35.063858+0000","last_peered":"2024-03-21T08:43:35.063858+0000","last_clean":"2024-03-21T08:43:35.063858+0000","last_became_active":"2024-03-21T08:43:34.076556+0000","last_became_peered":"2024-03-21T08:43:34.076556+0000","last_unstale":"2024-03-21T08:43:35.063858+0000","last_undegraded":"2024-03-21T08:43:35.063858+0000","last_fullsized":"2024-03-21T08:43:35.063858+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:03:33.956736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063663+0000","last_change":"2024-03-21T08:43:34.096375+0000","last_active":"2024-03-21T08:43:35.063663+0000","last_peered":"2024-03-21T08:43:35.063663+0000","last_clean":"2024-03-21T08:43:35.063663+0000","last_became_active":"2024-03-21T08:43:34.095943+0000","last_became_peered":"2024-03-21T08:43:34.095943+0000","last_unstale":"2024-03-21T08:43:35.063663+0000","last_undegraded":"2024-03-21T08:43:35.063663+0000","last_fullsized":"2024-03-21T08:43:35.063663+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:14:37.084279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065120+0000","last_change":"2024-03-21T08:43:34.075290+0000","last_active":"2024-03-21T08:43:35.065120+0000","last_peered":"2024-03-21T08:43:35.065120+0000","last_clean":"2024-03-21T08:43:35.065120+0000","last_became_active":"2024-03-21T08:43:34.074620+0000","last_became_peered":"2024-03-21T08:43:34.074620+0000","last_unstale":"2024-03-21T08:43:35.065120+0000","last_undegraded":"2024-03-21T08:43:35.065120+0000","last_fullsized":"2024-03-21T08:43:35.065120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:20:04.099452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.086057+0000","last_change":"2024-03-21T08:43:34.090738+0000","last_active":"2024-03-21T08:43:35.086057+0000","last_peered":"2024-03-21T08:43:35.086057+0000","last_clean":"2024-03-21T08:43:35.086057+0000","last_became_active":"2024-03-21T08:43:34.090493+0000","last_became_peered":"2024-03-21T08:43:34.090493+0000","last_unstale":"2024-03-21T08:43:35.086057+0000","last_undegraded":"2024-03-21T08:43:35.086057+0000","last_fullsized":"2024-03-21T08:43:35.086057+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:33:01.770928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265408+0000","last_change":"2024-03-21T08:43:34.101001+0000","last_active":"2024-03-21T08:43:35.265408+0000","last_peered":"2024-03-21T08:43:35.265408+0000","last_clean":"2024-03-21T08:43:35.265408+0000","last_became_active":"2024-03-21T08:43:34.100713+0000","last_became_peered":"2024-03-21T08:43:34.100713+0000","last_unstale":"2024-03-21T08:43:35.265408+0000","last_undegraded":"2024-03-21T08:43:35.265408+0000","last_fullsized":"2024-03-21T08:43:35.265408+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T08:56:36.030152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265532+0000","last_change":"2024-03-21T08:43:34.083354+0000","last_active":"2024-03-21T08:43:35.265532+0000","last_peered":"2024-03-21T08:43:35.265532+0000","last_clean":"2024-03-21T08:43:35.265532+0000","last_became_active":"2024-03-21T08:43:34.082948+0000","last_became_peered":"2024-03-21T08:43:34.082948+0000","last_unstale":"2024-03-21T08:43:35.265532+0000","last_undegraded":"2024-03-21T08:43:35.265532+0000","last_fullsized":"2024-03-21T08:43:35.265532+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:15:32.884893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083484+0000","last_change":"2024-03-21T08:43:34.088026+0000","last_active":"2024-03-21T08:43:35.083484+0000","last_peered":"2024-03-21T08:43:35.083484+0000","last_clean":"2024-03-21T08:43:35.083484+0000","last_became_active":"2024-03-21T08:43:34.087692+0000","last_became_peered":"2024-03-21T08:43:34.087692+0000","last_unstale":"2024-03-21T08:43:35.083484+0000","last_undegraded":"2024-03-21T08:43:35.083484+0000","last_fullsized":"2024-03-21T08:43:35.083484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:25.666989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267370+0000","last_change":"2024-03-21T08:43:34.098356+0000","last_active":"2024-03-21T08:43:35.267370+0000","last_peered":"2024-03-21T08:43:35.267370+0000","last_clean":"2024-03-21T08:43:35.267370+0000","last_became_active":"2024-03-21T08:43:34.098090+0000","last_became_peered":"2024-03-21T08:43:34.098090+0000","last_unstale":"2024-03-21T08:43:35.267370+0000","last_undegraded":"2024-03-21T08:43:35.267370+0000","last_fullsized":"2024-03-21T08:43:35.267370+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:52:55.221677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064585+0000","last_change":"2024-03-21T08:43:34.090757+0000","last_active":"2024-03-21T08:43:35.064585+0000","last_peered":"2024-03-21T08:43:35.064585+0000","last_clean":"2024-03-21T08:43:35.064585+0000","last_became_active":"2024-03-21T08:43:34.090125+0000","last_became_peered":"2024-03-21T08:43:34.090125+0000","last_unstale":"2024-03-21T08:43:35.064585+0000","last_undegraded":"2024-03-21T08:43:35.064585+0000","last_fullsized":"2024-03-21T08:43:35.064585+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:37:38.582943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266790+0000","last_change":"2024-03-21T08:43:34.099533+0000","last_active":"2024-03-21T08:43:35.266790+0000","last_peered":"2024-03-21T08:43:35.266790+0000","last_clean":"2024-03-21T08:43:35.266790+0000","last_became_active":"2024-03-21T08:43:34.099273+0000","last_became_peered":"2024-03-21T08:43:34.099273+0000","last_unstale":"2024-03-21T08:43:35.266790+0000","last_undegraded":"2024-03-21T08:43:35.266790+0000","last_fullsized":"2024-03-21T08:43:35.266790+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:08:17.054135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267033+0000","last_change":"2024-03-21T08:43:34.097269+0000","last_active":"2024-03-21T08:43:35.267033+0000","last_peered":"2024-03-21T08:43:35.267033+0000","last_clean":"2024-03-21T08:43:35.267033+0000","last_became_active":"2024-03-21T08:43:34.096445+0000","last_became_peered":"2024-03-21T08:43:34.096445+0000","last_unstale":"2024-03-21T08:43:35.267033+0000","last_undegraded":"2024-03-21T08:43:35.267033+0000","last_fullsized":"2024-03-21T08:43:35.267033+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:30:29.699066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064201+0000","last_change":"2024-03-21T08:43:34.069515+0000","last_active":"2024-03-21T08:43:35.064201+0000","last_peered":"2024-03-21T08:43:35.064201+0000","last_clean":"2024-03-21T08:43:35.064201+0000","last_became_active":"2024-03-21T08:43:34.069239+0000","last_became_peered":"2024-03-21T08:43:34.069239+0000","last_unstale":"2024-03-21T08:43:35.064201+0000","last_undegraded":"2024-03-21T08:43:35.064201+0000","last_fullsized":"2024-03-21T08:43:35.064201+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:43:32.501742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266593+0000","last_change":"2024-03-21T08:43:34.101974+0000","last_active":"2024-03-21T08:43:35.266593+0000","last_peered":"2024-03-21T08:43:35.266593+0000","last_clean":"2024-03-21T08:43:35.266593+0000","last_became_active":"2024-03-21T08:43:34.101668+0000","last_became_peered":"2024-03-21T08:43:34.101668+0000","last_unstale":"2024-03-21T08:43:35.266593+0000","last_undegraded":"2024-03-21T08:43:35.266593+0000","last_fullsized":"2024-03-21T08:43:35.266593+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:57:05.618262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063933+0000","last_change":"2024-03-21T08:43:34.097093+0000","last_active":"2024-03-21T08:43:35.063933+0000","last_peered":"2024-03-21T08:43:35.063933+0000","last_clean":"2024-03-21T08:43:35.063933+0000","last_became_active":"2024-03-21T08:43:34.096448+0000","last_became_peered":"2024-03-21T08:43:34.096448+0000","last_unstale":"2024-03-21T08:43:35.063933+0000","last_undegraded":"2024-03-21T08:43:35.063933+0000","last_fullsized":"2024-03-21T08:43:35.063933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:38:55.328473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.085869+0000","last_change":"2024-03-21T08:43:34.089864+0000","last_active":"2024-03-21T08:43:35.085869+0000","last_peered":"2024-03-21T08:43:35.085869+0000","last_clean":"2024-03-21T08:43:35.085869+0000","last_became_active":"2024-03-21T08:43:34.089605+0000","last_became_peered":"2024-03-21T08:43:34.089605+0000","last_unstale":"2024-03-21T08:43:35.085869+0000","last_undegraded":"2024-03-21T08:43:35.085869+0000","last_fullsized":"2024-03-21T08:43:35.085869+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:22:14.199100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266969+0000","last_change":"2024-03-21T08:43:34.070708+0000","last_active":"2024-03-21T08:43:35.266969+0000","last_peered":"2024-03-21T08:43:35.266969+0000","last_clean":"2024-03-21T08:43:35.266969+0000","last_became_active":"2024-03-21T08:43:34.070458+0000","last_became_peered":"2024-03-21T08:43:34.070458+0000","last_unstale":"2024-03-21T08:43:35.266969+0000","last_undegraded":"2024-03-21T08:43:35.266969+0000","last_fullsized":"2024-03-21T08:43:35.266969+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:29:07.630754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065136+0000","last_change":"2024-03-21T08:43:34.084818+0000","last_active":"2024-03-21T08:43:35.065136+0000","last_peered":"2024-03-21T08:43:35.065136+0000","last_clean":"2024-03-21T08:43:35.065136+0000","last_became_active":"2024-03-21T08:43:34.084490+0000","last_became_peered":"2024-03-21T08:43:34.084490+0000","last_unstale":"2024-03-21T08:43:35.065136+0000","last_undegraded":"2024-03-21T08:43:35.065136+0000","last_fullsized":"2024-03-21T08:43:35.065136+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:13:14.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083917+0000","last_change":"2024-03-21T08:43:34.082575+0000","last_active":"2024-03-21T08:43:35.083917+0000","last_peered":"2024-03-21T08:43:35.083917+0000","last_clean":"2024-03-21T08:43:35.083917+0000","last_became_active":"2024-03-21T08:43:34.082126+0000","last_became_peered":"2024-03-21T08:43:34.082126+0000","last_unstale":"2024-03-21T08:43:35.083917+0000","last_undegraded":"2024-03-21T08:43:35.083917+0000","last_fullsized":"2024-03-21T08:43:35.083917+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:03.252966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266745+0000","last_change":"2024-03-21T08:43:34.074171+0000","last_active":"2024-03-21T08:43:35.266745+0000","last_peered":"2024-03-21T08:43:35.266745+0000","last_clean":"2024-03-21T08:43:35.266745+0000","last_became_active":"2024-03-21T08:43:34.073918+0000","last_became_peered":"2024-03-21T08:43:34.073918+0000","last_unstale":"2024-03-21T08:43:35.266745+0000","last_undegraded":"2024-03-21T08:43:35.266745+0000","last_fullsized":"2024-03-21T08:43:35.266745+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:24:56.793423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266980+0000","last_change":"2024-03-21T08:43:34.086013+0000","last_active":"2024-03-21T08:43:35.266980+0000","last_peered":"2024-03-21T08:43:35.266980+0000","last_clean":"2024-03-21T08:43:35.266980+0000","last_became_active":"2024-03-21T08:43:34.082821+0000","last_became_peered":"2024-03-21T08:43:34.082821+0000","last_unstale":"2024-03-21T08:43:35.266980+0000","last_undegraded":"2024-03-21T08:43:35.266980+0000","last_fullsized":"2024-03-21T08:43:35.266980+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:44:13.212743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064007+0000","last_change":"2024-03-21T08:43:34.079738+0000","last_active":"2024-03-21T08:43:35.064007+0000","last_peered":"2024-03-21T08:43:35.064007+0000","last_clean":"2024-03-21T08:43:35.064007+0000","last_became_active":"2024-03-21T08:43:34.079479+0000","last_became_peered":"2024-03-21T08:43:34.079479+0000","last_unstale":"2024-03-21T08:43:35.064007+0000","last_undegraded":"2024-03-21T08:43:35.064007+0000","last_fullsized":"2024-03-21T08:43:35.064007+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:27.442520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266058+0000","last_change":"2024-03-21T08:43:34.069752+0000","last_active":"2024-03-21T08:43:35.266058+0000","last_peered":"2024-03-21T08:43:35.266058+0000","last_clean":"2024-03-21T08:43:35.266058+0000","last_became_active":"2024-03-21T08:43:34.068765+0000","last_became_peered":"2024-03-21T08:43:34.068765+0000","last_unstale":"2024-03-21T08:43:35.266058+0000","last_undegraded":"2024-03-21T08:43:35.266058+0000","last_fullsized":"2024-03-21T08:43:35.266058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:19:50.235185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083560+0000","last_change":"2024-03-21T08:43:34.092372+0000","last_active":"2024-03-21T08:43:35.083560+0000","last_peered":"2024-03-21T08:43:35.083560+0000","last_clean":"2024-03-21T08:43:35.083560+0000","last_became_active":"2024-03-21T08:43:34.092101+0000","last_became_peered":"2024-03-21T08:43:34.092101+0000","last_unstale":"2024-03-21T08:43:35.083560+0000","last_undegraded":"2024-03-21T08:43:35.083560+0000","last_fullsized":"2024-03-21T08:43:35.083560+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:13:14.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084006+0000","last_change":"2024-03-21T08:43:34.087154+0000","last_active":"2024-03-21T08:43:35.084006+0000","last_peered":"2024-03-21T08:43:35.084006+0000","last_clean":"2024-03-21T08:43:35.084006+0000","last_became_active":"2024-03-21T08:43:34.086887+0000","last_became_peered":"2024-03-21T08:43:34.086887+0000","last_unstale":"2024-03-21T08:43:35.084006+0000","last_undegraded":"2024-03-21T08:43:35.084006+0000","last_fullsized":"2024-03-21T08:43:35.084006+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:33:07.228390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"53'4","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084217+0000","last_change":"2024-03-21T08:43:33.081655+0000","last_active":"2024-03-21T08:43:35.084217+0000","last_peered":"2024-03-21T08:43:35.084217+0000","last_clean":"2024-03-21T08:43:35.084217+0000","last_became_active":"2024-03-21T08:43:33.081260+0000","last_became_peered":"2024-03-21T08:43:33.081260+0000","last_unstale":"2024-03-21T08:43:35.084217+0000","last_undegraded":"2024-03-21T08:43:35.084217+0000","last_fullsized":"2024-03-21T08:43:35.084217+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063939+0000","last_change":"2024-03-21T08:43:34.070772+0000","last_active":"2024-03-21T08:43:35.063939+0000","last_peered":"2024-03-21T08:43:35.063939+0000","last_clean":"2024-03-21T08:43:35.063939+0000","last_became_active":"2024-03-21T08:43:34.070484+0000","last_became_peered":"2024-03-21T08:43:34.070484+0000","last_unstale":"2024-03-21T08:43:35.063939+0000","last_undegraded":"2024-03-21T08:43:35.063939+0000","last_fullsized":"2024-03-21T08:43:35.063939+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:53:30.468809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267165+0000","last_change":"2024-03-21T08:43:33.078482+0000","last_active":"2024-03-21T08:43:35.267165+0000","last_peered":"2024-03-21T08:43:35.267165+0000","last_clean":"2024-03-21T08:43:35.267165+0000","last_became_active":"2024-03-21T08:43:33.078221+0000","last_became_peered":"2024-03-21T08:43:33.078221+0000","last_unstale":"2024-03-21T08:43:35.267165+0000","last_undegraded":"2024-03-21T08:43:35.267165+0000","last_fullsized":"2024-03-21T08:43:35.267165+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:52:15.693236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084200+0000","last_change":"2024-03-21T08:43:34.086549+0000","last_active":"2024-03-21T08:43:35.084200+0000","last_peered":"2024-03-21T08:43:35.084200+0000","last_clean":"2024-03-21T08:43:35.084200+0000","last_became_active":"2024-03-21T08:43:34.086138+0000","last_became_peered":"2024-03-21T08:43:34.086138+0000","last_unstale":"2024-03-21T08:43:35.084200+0000","last_undegraded":"2024-03-21T08:43:35.084200+0000","last_fullsized":"2024-03-21T08:43:35.084200+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:28:55.959297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084084+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:35.084084+0000","last_peered":"2024-03-21T08:43:35.084084+0000","last_clean":"2024-03-21T08:43:35.084084+0000","last_became_active":"2024-03-21T08:43:33.080225+0000","last_became_peered":"2024-03-21T08:43:33.080225+0000","last_unstale":"2024-03-21T08:43:35.084084+0000","last_undegraded":"2024-03-21T08:43:35.084084+0000","last_fullsized":"2024-03-21T08:43:35.084084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083978+0000","last_change":"2024-03-21T08:43:34.087074+0000","last_active":"2024-03-21T08:43:35.083978+0000","last_peered":"2024-03-21T08:43:35.083978+0000","last_clean":"2024-03-21T08:43:35.083978+0000","last_became_active":"2024-03-21T08:43:34.086751+0000","last_became_peered":"2024-03-21T08:43:34.086751+0000","last_unstale":"2024-03-21T08:43:35.083978+0000","last_undegraded":"2024-03-21T08:43:35.083978+0000","last_fullsized":"2024-03-21T08:43:35.083978+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065338+0000","last_change":"2024-03-21T08:43:33.083210+0000","last_active":"2024-03-21T08:43:35.065338+0000","last_peered":"2024-03-21T08:43:35.065338+0000","last_clean":"2024-03-21T08:43:35.065338+0000","last_became_active":"2024-03-21T08:43:33.082822+0000","last_became_peered":"2024-03-21T08:43:33.082822+0000","last_unstale":"2024-03-21T08:43:35.065338+0000","last_undegraded":"2024-03-21T08:43:35.065338+0000","last_fullsized":"2024-03-21T08:43:35.065338+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266331+0000","last_change":"2024-03-21T08:43:34.087514+0000","last_active":"2024-03-21T08:43:35.266331+0000","last_peered":"2024-03-21T08:43:35.266331+0000","last_clean":"2024-03-21T08:43:35.266331+0000","last_became_active":"2024-03-21T08:43:34.087212+0000","last_became_peered":"2024-03-21T08:43:34.087212+0000","last_unstale":"2024-03-21T08:43:35.266331+0000","last_undegraded":"2024-03-21T08:43:35.266331+0000","last_fullsized":"2024-03-21T08:43:35.266331+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:33:07.228390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265371+0000","last_change":"2024-03-21T08:43:33.078204+0000","last_active":"2024-03-21T08:43:35.265371+0000","last_peered":"2024-03-21T08:43:35.265371+0000","last_clean":"2024-03-21T08:43:35.265371+0000","last_became_active":"2024-03-21T08:43:33.077529+0000","last_became_peered":"2024-03-21T08:43:33.077529+0000","last_unstale":"2024-03-21T08:43:35.265371+0000","last_undegraded":"2024-03-21T08:43:35.265371+0000","last_fullsized":"2024-03-21T08:43:35.265371+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:11:04.693260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266690+0000","last_change":"2024-03-21T08:43:34.099085+0000","last_active":"2024-03-21T08:43:35.266690+0000","last_peered":"2024-03-21T08:43:35.266690+0000","last_clean":"2024-03-21T08:43:35.266690+0000","last_became_active":"2024-03-21T08:43:34.096614+0000","last_became_peered":"2024-03-21T08:43:34.096614+0000","last_unstale":"2024-03-21T08:43:35.266690+0000","last_undegraded":"2024-03-21T08:43:35.266690+0000","last_fullsized":"2024-03-21T08:43:35.266690+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:26:33.567206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083563+0000","last_change":"2024-03-21T08:43:33.078883+0000","last_active":"2024-03-21T08:43:35.083563+0000","last_peered":"2024-03-21T08:43:35.083563+0000","last_clean":"2024-03-21T08:43:35.083563+0000","last_became_active":"2024-03-21T08:43:33.078618+0000","last_became_peered":"2024-03-21T08:43:33.078618+0000","last_unstale":"2024-03-21T08:43:35.083563+0000","last_undegraded":"2024-03-21T08:43:35.083563+0000","last_fullsized":"2024-03-21T08:43:35.083563+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:48:59.806440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.086013+0000","last_change":"2024-03-21T08:43:34.082546+0000","last_active":"2024-03-21T08:43:35.086013+0000","last_peered":"2024-03-21T08:43:35.086013+0000","last_clean":"2024-03-21T08:43:35.086013+0000","last_became_active":"2024-03-21T08:43:34.082250+0000","last_became_peered":"2024-03-21T08:43:34.082250+0000","last_unstale":"2024-03-21T08:43:35.086013+0000","last_undegraded":"2024-03-21T08:43:35.086013+0000","last_fullsized":"2024-03-21T08:43:35.086013+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:58:25.147246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064603+0000","last_change":"2024-03-21T08:43:33.075815+0000","last_active":"2024-03-21T08:43:35.064603+0000","last_peered":"2024-03-21T08:43:35.064603+0000","last_clean":"2024-03-21T08:43:35.064603+0000","last_became_active":"2024-03-21T08:43:33.075533+0000","last_became_peered":"2024-03-21T08:43:33.075533+0000","last_unstale":"2024-03-21T08:43:35.064603+0000","last_undegraded":"2024-03-21T08:43:35.064603+0000","last_fullsized":"2024-03-21T08:43:35.064603+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266477+0000","last_change":"2024-03-21T08:43:34.086730+0000","last_active":"2024-03-21T08:43:35.266477+0000","last_peered":"2024-03-21T08:43:35.266477+0000","last_clean":"2024-03-21T08:43:35.266477+0000","last_became_active":"2024-03-21T08:43:34.086471+0000","last_became_peered":"2024-03-21T08:43:34.086471+0000","last_unstale":"2024-03-21T08:43:35.266477+0000","last_undegraded":"2024-03-21T08:43:35.266477+0000","last_fullsized":"2024-03-21T08:43:35.266477+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:00:52.228139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265568+0000","last_change":"2024-03-21T08:43:33.076297+0000","last_active":"2024-03-21T08:43:35.265568+0000","last_peered":"2024-03-21T08:43:35.265568+0000","last_clean":"2024-03-21T08:43:35.265568+0000","last_became_active":"2024-03-21T08:43:33.075418+0000","last_became_peered":"2024-03-21T08:43:33.075418+0000","last_unstale":"2024-03-21T08:43:35.265568+0000","last_undegraded":"2024-03-21T08:43:35.265568+0000","last_fullsized":"2024-03-21T08:43:35.265568+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:20:51.363983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265878+0000","last_change":"2024-03-21T08:43:34.083842+0000","last_active":"2024-03-21T08:43:35.265878+0000","last_peered":"2024-03-21T08:43:35.265878+0000","last_clean":"2024-03-21T08:43:35.265878+0000","last_became_active":"2024-03-21T08:43:34.083499+0000","last_became_peered":"2024-03-21T08:43:34.083499+0000","last_unstale":"2024-03-21T08:43:35.265878+0000","last_undegraded":"2024-03-21T08:43:35.265878+0000","last_fullsized":"2024-03-21T08:43:35.265878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:11:42.706198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064452+0000","last_change":"2024-03-21T08:43:33.084904+0000","last_active":"2024-03-21T08:43:35.064452+0000","last_peered":"2024-03-21T08:43:35.064452+0000","last_clean":"2024-03-21T08:43:35.064452+0000","last_became_active":"2024-03-21T08:43:33.084490+0000","last_became_peered":"2024-03-21T08:43:33.084490+0000","last_unstale":"2024-03-21T08:43:35.064452+0000","last_undegraded":"2024-03-21T08:43:35.064452+0000","last_fullsized":"2024-03-21T08:43:35.064452+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:31:59.934028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064343+0000","last_change":"2024-03-21T08:43:34.073629+0000","last_active":"2024-03-21T08:43:35.064343+0000","last_peered":"2024-03-21T08:43:35.064343+0000","last_clean":"2024-03-21T08:43:35.064343+0000","last_became_active":"2024-03-21T08:43:34.073264+0000","last_became_peered":"2024-03-21T08:43:34.073264+0000","last_unstale":"2024-03-21T08:43:35.064343+0000","last_undegraded":"2024-03-21T08:43:35.064343+0000","last_fullsized":"2024-03-21T08:43:35.064343+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:30:48.719653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267268+0000","last_change":"2024-03-21T08:43:33.080472+0000","last_active":"2024-03-21T08:43:35.267268+0000","last_peered":"2024-03-21T08:43:35.267268+0000","last_clean":"2024-03-21T08:43:35.267268+0000","last_became_active":"2024-03-21T08:43:33.080295+0000","last_became_peered":"2024-03-21T08:43:33.080295+0000","last_unstale":"2024-03-21T08:43:35.267268+0000","last_undegraded":"2024-03-21T08:43:35.267268+0000","last_fullsized":"2024-03-21T08:43:35.267268+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:03:50.003394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063425+0000","last_change":"2024-03-21T08:43:34.082659+0000","last_active":"2024-03-21T08:43:35.063425+0000","last_peered":"2024-03-21T08:43:35.063425+0000","last_clean":"2024-03-21T08:43:35.063425+0000","last_became_active":"2024-03-21T08:43:34.082297+0000","last_became_peered":"2024-03-21T08:43:34.082297+0000","last_unstale":"2024-03-21T08:43:35.063425+0000","last_undegraded":"2024-03-21T08:43:35.063425+0000","last_fullsized":"2024-03-21T08:43:35.063425+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:10.925949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"53'3","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064787+0000","last_change":"2024-03-21T08:43:33.079062+0000","last_active":"2024-03-21T08:43:35.064787+0000","last_peered":"2024-03-21T08:43:35.064787+0000","last_clean":"2024-03-21T08:43:35.064787+0000","last_became_active":"2024-03-21T08:43:33.078759+0000","last_became_peered":"2024-03-21T08:43:33.078759+0000","last_unstale":"2024-03-21T08:43:35.064787+0000","last_undegraded":"2024-03-21T08:43:35.064787+0000","last_fullsized":"2024-03-21T08:43:35.064787+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:42:09.919722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064785+0000","last_change":"2024-03-21T08:43:34.074492+0000","last_active":"2024-03-21T08:43:35.064785+0000","last_peered":"2024-03-21T08:43:35.064785+0000","last_clean":"2024-03-21T08:43:35.064785+0000","last_became_active":"2024-03-21T08:43:34.074094+0000","last_became_peered":"2024-03-21T08:43:34.074094+0000","last_unstale":"2024-03-21T08:43:35.064785+0000","last_undegraded":"2024-03-21T08:43:35.064785+0000","last_fullsized":"2024-03-21T08:43:35.064785+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:35:04.089313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"53'6","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265735+0000","last_change":"2024-03-21T08:43:33.074842+0000","last_active":"2024-03-21T08:43:35.265735+0000","last_peered":"2024-03-21T08:43:35.265735+0000","last_clean":"2024-03-21T08:43:35.265735+0000","last_became_active":"2024-03-21T08:43:33.074382+0000","last_became_peered":"2024-03-21T08:43:33.074382+0000","last_unstale":"2024-03-21T08:43:35.265735+0000","last_undegraded":"2024-03-21T08:43:35.265735+0000","last_fullsized":"2024-03-21T08:43:35.265735+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:55:17.408018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064825+0000","last_change":"2024-03-21T08:43:34.070945+0000","last_active":"2024-03-21T08:43:35.064825+0000","last_peered":"2024-03-21T08:43:35.064825+0000","last_clean":"2024-03-21T08:43:35.064825+0000","last_became_active":"2024-03-21T08:43:34.070613+0000","last_became_peered":"2024-03-21T08:43:34.070613+0000","last_unstale":"2024-03-21T08:43:35.064825+0000","last_undegraded":"2024-03-21T08:43:35.064825+0000","last_fullsized":"2024-03-21T08:43:35.064825+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:31:37.489435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266714+0000","last_change":"2024-03-21T08:43:33.080488+0000","last_active":"2024-03-21T08:43:35.266714+0000","last_peered":"2024-03-21T08:43:35.266714+0000","last_clean":"2024-03-21T08:43:35.266714+0000","last_became_active":"2024-03-21T08:43:33.080224+0000","last_became_peered":"2024-03-21T08:43:33.080224+0000","last_unstale":"2024-03-21T08:43:35.266714+0000","last_undegraded":"2024-03-21T08:43:35.266714+0000","last_fullsized":"2024-03-21T08:43:35.266714+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064663+0000","last_change":"2024-03-21T08:43:34.078879+0000","last_active":"2024-03-21T08:43:35.064663+0000","last_peered":"2024-03-21T08:43:35.064663+0000","last_clean":"2024-03-21T08:43:35.064663+0000","last_became_active":"2024-03-21T08:43:34.078610+0000","last_became_peered":"2024-03-21T08:43:34.078610+0000","last_unstale":"2024-03-21T08:43:35.064663+0000","last_undegraded":"2024-03-21T08:43:35.064663+0000","last_fullsized":"2024-03-21T08:43:35.064663+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:06:13.882644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.085875+0000","last_change":"2024-03-21T08:43:33.085060+0000","last_active":"2024-03-21T08:43:35.085875+0000","last_peered":"2024-03-21T08:43:35.085875+0000","last_clean":"2024-03-21T08:43:35.085875+0000","last_became_active":"2024-03-21T08:43:33.084725+0000","last_became_peered":"2024-03-21T08:43:33.084725+0000","last_unstale":"2024-03-21T08:43:35.085875+0000","last_undegraded":"2024-03-21T08:43:35.085875+0000","last_fullsized":"2024-03-21T08:43:35.085875+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265565+0000","last_change":"2024-03-21T08:43:34.085498+0000","last_active":"2024-03-21T08:43:35.265565+0000","last_peered":"2024-03-21T08:43:35.265565+0000","last_clean":"2024-03-21T08:43:35.265565+0000","last_became_active":"2024-03-21T08:43:34.085185+0000","last_became_peered":"2024-03-21T08:43:34.085185+0000","last_unstale":"2024-03-21T08:43:35.265565+0000","last_undegraded":"2024-03-21T08:43:35.265565+0000","last_fullsized":"2024-03-21T08:43:35.265565+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:41.809827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064697+0000","last_change":"2024-03-21T08:43:33.075978+0000","last_active":"2024-03-21T08:43:35.064697+0000","last_peered":"2024-03-21T08:43:35.064697+0000","last_clean":"2024-03-21T08:43:35.064697+0000","last_became_active":"2024-03-21T08:43:33.075230+0000","last_became_peered":"2024-03-21T08:43:33.075230+0000","last_unstale":"2024-03-21T08:43:35.064697+0000","last_undegraded":"2024-03-21T08:43:35.064697+0000","last_fullsized":"2024-03-21T08:43:35.064697+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064114+0000","last_change":"2024-03-21T08:43:34.081971+0000","last_active":"2024-03-21T08:43:35.064114+0000","last_peered":"2024-03-21T08:43:35.064114+0000","last_clean":"2024-03-21T08:43:35.064114+0000","last_became_active":"2024-03-21T08:43:34.080960+0000","last_became_peered":"2024-03-21T08:43:34.080960+0000","last_unstale":"2024-03-21T08:43:35.064114+0000","last_undegraded":"2024-03-21T08:43:35.064114+0000","last_fullsized":"2024-03-21T08:43:35.064114+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:21:57.382333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"53'8","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063857+0000","last_change":"2024-03-21T08:43:33.082199+0000","last_active":"2024-03-21T08:43:35.063857+0000","last_peered":"2024-03-21T08:43:35.063857+0000","last_clean":"2024-03-21T08:43:35.063857+0000","last_became_active":"2024-03-21T08:43:33.081892+0000","last_became_peered":"2024-03-21T08:43:33.081892+0000","last_unstale":"2024-03-21T08:43:35.063857+0000","last_undegraded":"2024-03-21T08:43:35.063857+0000","last_fullsized":"2024-03-21T08:43:35.063857+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:42:19.722045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064758+0000","last_change":"2024-03-21T08:43:34.086279+0000","last_active":"2024-03-21T08:43:35.064758+0000","last_peered":"2024-03-21T08:43:35.064758+0000","last_clean":"2024-03-21T08:43:35.064758+0000","last_became_active":"2024-03-21T08:43:34.086021+0000","last_became_peered":"2024-03-21T08:43:34.086021+0000","last_unstale":"2024-03-21T08:43:35.064758+0000","last_undegraded":"2024-03-21T08:43:35.064758+0000","last_fullsized":"2024-03-21T08:43:35.064758+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:06:30.074732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267343+0000","last_change":"2024-03-21T08:43:33.082173+0000","last_active":"2024-03-21T08:43:35.267343+0000","last_peered":"2024-03-21T08:43:35.267343+0000","last_clean":"2024-03-21T08:43:35.267343+0000","last_became_active":"2024-03-21T08:43:33.081902+0000","last_became_peered":"2024-03-21T08:43:33.081902+0000","last_unstale":"2024-03-21T08:43:35.267343+0000","last_undegraded":"2024-03-21T08:43:35.267343+0000","last_fullsized":"2024-03-21T08:43:35.267343+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T11:29:40.400581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065436+0000","last_change":"2024-03-21T08:43:34.070653+0000","last_active":"2024-03-21T08:43:35.065436+0000","last_peered":"2024-03-21T08:43:35.065436+0000","last_clean":"2024-03-21T08:43:35.065436+0000","last_became_active":"2024-03-21T08:43:34.070064+0000","last_became_peered":"2024-03-21T08:43:34.070064+0000","last_unstale":"2024-03-21T08:43:35.065436+0000","last_undegraded":"2024-03-21T08:43:35.065436+0000","last_fullsized":"2024-03-21T08:43:35.065436+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:59:03.252966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084043+0000","last_change":"2024-03-21T08:43:33.077524+0000","last_active":"2024-03-21T08:43:35.084043+0000","last_peered":"2024-03-21T08:43:35.084043+0000","last_clean":"2024-03-21T08:43:35.084043+0000","last_became_active":"2024-03-21T08:43:33.077163+0000","last_became_peered":"2024-03-21T08:43:33.077163+0000","last_unstale":"2024-03-21T08:43:35.084043+0000","last_undegraded":"2024-03-21T08:43:35.084043+0000","last_fullsized":"2024-03-21T08:43:35.084043+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.267185+0000","last_change":"2024-03-21T08:43:34.088073+0000","last_active":"2024-03-21T08:43:35.267185+0000","last_peered":"2024-03-21T08:43:35.267185+0000","last_clean":"2024-03-21T08:43:35.267185+0000","last_became_active":"2024-03-21T08:43:34.087799+0000","last_became_peered":"2024-03-21T08:43:34.087799+0000","last_unstale":"2024-03-21T08:43:35.267185+0000","last_undegraded":"2024-03-21T08:43:35.267185+0000","last_fullsized":"2024-03-21T08:43:35.267185+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:45:13.878888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065165+0000","last_change":"2024-03-21T08:43:33.078372+0000","last_active":"2024-03-21T08:43:35.065165+0000","last_peered":"2024-03-21T08:43:35.065165+0000","last_clean":"2024-03-21T08:43:35.065165+0000","last_became_active":"2024-03-21T08:43:33.077797+0000","last_became_peered":"2024-03-21T08:43:33.077797+0000","last_unstale":"2024-03-21T08:43:35.065165+0000","last_undegraded":"2024-03-21T08:43:35.065165+0000","last_fullsized":"2024-03-21T08:43:35.065165+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:16:56.121851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064636+0000","last_change":"2024-03-21T08:43:34.072036+0000","last_active":"2024-03-21T08:43:35.064636+0000","last_peered":"2024-03-21T08:43:35.064636+0000","last_clean":"2024-03-21T08:43:35.064636+0000","last_became_active":"2024-03-21T08:43:34.071710+0000","last_became_peered":"2024-03-21T08:43:34.071710+0000","last_unstale":"2024-03-21T08:43:35.064636+0000","last_undegraded":"2024-03-21T08:43:35.064636+0000","last_fullsized":"2024-03-21T08:43:35.064636+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:01.482531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083881+0000","last_change":"2024-03-21T08:43:33.079120+0000","last_active":"2024-03-21T08:43:35.083881+0000","last_peered":"2024-03-21T08:43:35.083881+0000","last_clean":"2024-03-21T08:43:35.083881+0000","last_became_active":"2024-03-21T08:43:33.078672+0000","last_became_peered":"2024-03-21T08:43:33.078672+0000","last_unstale":"2024-03-21T08:43:35.083881+0000","last_undegraded":"2024-03-21T08:43:35.083881+0000","last_fullsized":"2024-03-21T08:43:35.083881+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064890+0000","last_change":"2024-03-21T08:43:34.073220+0000","last_active":"2024-03-21T08:43:35.064890+0000","last_peered":"2024-03-21T08:43:35.064890+0000","last_clean":"2024-03-21T08:43:35.064890+0000","last_became_active":"2024-03-21T08:43:34.072909+0000","last_became_peered":"2024-03-21T08:43:34.072909+0000","last_unstale":"2024-03-21T08:43:35.064890+0000","last_undegraded":"2024-03-21T08:43:35.064890+0000","last_fullsized":"2024-03-21T08:43:35.064890+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T20:37:25.666989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266922+0000","last_change":"2024-03-21T08:43:33.074311+0000","last_active":"2024-03-21T08:43:35.266922+0000","last_peered":"2024-03-21T08:43:35.266922+0000","last_clean":"2024-03-21T08:43:35.266922+0000","last_became_active":"2024-03-21T08:43:33.074047+0000","last_became_peered":"2024-03-21T08:43:33.074047+0000","last_unstale":"2024-03-21T08:43:35.266922+0000","last_undegraded":"2024-03-21T08:43:35.266922+0000","last_fullsized":"2024-03-21T08:43:35.266922+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:18:42.856030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083835+0000","last_change":"2024-03-21T08:43:34.090009+0000","last_active":"2024-03-21T08:43:35.083835+0000","last_peered":"2024-03-21T08:43:35.083835+0000","last_clean":"2024-03-21T08:43:35.083835+0000","last_became_active":"2024-03-21T08:43:34.089356+0000","last_became_peered":"2024-03-21T08:43:34.089356+0000","last_unstale":"2024-03-21T08:43:35.083835+0000","last_undegraded":"2024-03-21T08:43:35.083835+0000","last_fullsized":"2024-03-21T08:43:35.083835+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:20:04.245491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063389+0000","last_change":"2024-03-21T08:43:33.080008+0000","last_active":"2024-03-21T08:43:35.063389+0000","last_peered":"2024-03-21T08:43:35.063389+0000","last_clean":"2024-03-21T08:43:35.063389+0000","last_became_active":"2024-03-21T08:43:33.079744+0000","last_became_peered":"2024-03-21T08:43:33.079744+0000","last_unstale":"2024-03-21T08:43:35.063389+0000","last_undegraded":"2024-03-21T08:43:35.063389+0000","last_fullsized":"2024-03-21T08:43:35.063389+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:43:34.225900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.083515+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:35.083515+0000","last_peered":"2024-03-21T08:43:35.083515+0000","last_clean":"2024-03-21T08:43:35.083515+0000","last_became_active":"2024-03-21T08:43:34.089398+0000","last_became_peered":"2024-03-21T08:43:34.089398+0000","last_unstale":"2024-03-21T08:43:35.083515+0000","last_undegraded":"2024-03-21T08:43:35.083515+0000","last_fullsized":"2024-03-21T08:43:35.083515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:20:04.099452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064150+0000","last_change":"2024-03-21T08:43:33.076251+0000","last_active":"2024-03-21T08:43:35.064150+0000","last_peered":"2024-03-21T08:43:35.064150+0000","last_clean":"2024-03-21T08:43:35.064150+0000","last_became_active":"2024-03-21T08:43:33.074661+0000","last_became_peered":"2024-03-21T08:43:33.074661+0000","last_unstale":"2024-03-21T08:43:35.064150+0000","last_undegraded":"2024-03-21T08:43:35.064150+0000","last_fullsized":"2024-03-21T08:43:35.064150+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065344+0000","last_change":"2024-03-21T08:43:34.068766+0000","last_active":"2024-03-21T08:43:35.065344+0000","last_peered":"2024-03-21T08:43:35.065344+0000","last_clean":"2024-03-21T08:43:35.065344+0000","last_became_active":"2024-03-21T08:43:34.068447+0000","last_became_peered":"2024-03-21T08:43:34.068447+0000","last_unstale":"2024-03-21T08:43:35.065344+0000","last_undegraded":"2024-03-21T08:43:35.065344+0000","last_fullsized":"2024-03-21T08:43:35.065344+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:44:52.985325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063937+0000","last_change":"2024-03-21T08:43:33.079309+0000","last_active":"2024-03-21T08:43:35.063937+0000","last_peered":"2024-03-21T08:43:35.063937+0000","last_clean":"2024-03-21T08:43:35.063937+0000","last_became_active":"2024-03-21T08:43:33.079038+0000","last_became_peered":"2024-03-21T08:43:33.079038+0000","last_unstale":"2024-03-21T08:43:35.063937+0000","last_undegraded":"2024-03-21T08:43:35.063937+0000","last_fullsized":"2024-03-21T08:43:35.063937+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:48:23.361743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065497+0000","last_change":"2024-03-21T08:43:34.079196+0000","last_active":"2024-03-21T08:43:35.065497+0000","last_peered":"2024-03-21T08:43:35.065497+0000","last_clean":"2024-03-21T08:43:35.065497+0000","last_became_active":"2024-03-21T08:43:34.078900+0000","last_became_peered":"2024-03-21T08:43:34.078900+0000","last_unstale":"2024-03-21T08:43:35.065497+0000","last_undegraded":"2024-03-21T08:43:35.065497+0000","last_fullsized":"2024-03-21T08:43:35.065497+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"54'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084143+0000","last_change":"2024-03-21T08:43:33.078636+0000","last_active":"2024-03-21T08:43:35.084143+0000","last_peered":"2024-03-21T08:43:35.084143+0000","last_clean":"2024-03-21T08:43:35.084143+0000","last_became_active":"2024-03-21T08:43:33.078336+0000","last_became_peered":"2024-03-21T08:43:33.078336+0000","last_unstale":"2024-03-21T08:43:35.084143+0000","last_undegraded":"2024-03-21T08:43:35.084143+0000","last_fullsized":"2024-03-21T08:43:35.084143+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T18:05:57.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063573+0000","last_change":"2024-03-21T08:43:34.082469+0000","last_active":"2024-03-21T08:43:35.063573+0000","last_peered":"2024-03-21T08:43:35.063573+0000","last_clean":"2024-03-21T08:43:35.063573+0000","last_became_active":"2024-03-21T08:43:34.082227+0000","last_became_peered":"2024-03-21T08:43:34.082227+0000","last_unstale":"2024-03-21T08:43:35.063573+0000","last_undegraded":"2024-03-21T08:43:35.063573+0000","last_fullsized":"2024-03-21T08:43:35.063573+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:40:59.946986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.085958+0000","last_change":"2024-03-21T08:43:33.077725+0000","last_active":"2024-03-21T08:43:35.085958+0000","last_peered":"2024-03-21T08:43:35.085958+0000","last_clean":"2024-03-21T08:43:35.085958+0000","last_became_active":"2024-03-21T08:43:33.077452+0000","last_became_peered":"2024-03-21T08:43:33.077452+0000","last_unstale":"2024-03-21T08:43:35.085958+0000","last_undegraded":"2024-03-21T08:43:35.085958+0000","last_fullsized":"2024-03-21T08:43:35.085958+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:22:33.956371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.084098+0000","last_change":"2024-03-21T08:43:34.082367+0000","last_active":"2024-03-21T08:43:35.084098+0000","last_peered":"2024-03-21T08:43:35.084098+0000","last_clean":"2024-03-21T08:43:35.084098+0000","last_became_active":"2024-03-21T08:43:34.081770+0000","last_became_peered":"2024-03-21T08:43:34.081770+0000","last_unstale":"2024-03-21T08:43:35.084098+0000","last_undegraded":"2024-03-21T08:43:35.084098+0000","last_fullsized":"2024-03-21T08:43:35.084098+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:31:03.090585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.065464+0000","last_change":"2024-03-21T08:43:33.076652+0000","last_active":"2024-03-21T08:43:35.065464+0000","last_peered":"2024-03-21T08:43:35.065464+0000","last_clean":"2024-03-21T08:43:35.065464+0000","last_became_active":"2024-03-21T08:43:33.074101+0000","last_became_peered":"2024-03-21T08:43:33.074101+0000","last_unstale":"2024-03-21T08:43:35.065464+0000","last_undegraded":"2024-03-21T08:43:35.065464+0000","last_fullsized":"2024-03-21T08:43:35.065464+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:44:32.707151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266463+0000","last_change":"2024-03-21T08:43:34.087659+0000","last_active":"2024-03-21T08:43:35.266463+0000","last_peered":"2024-03-21T08:43:35.266463+0000","last_clean":"2024-03-21T08:43:35.266463+0000","last_became_active":"2024-03-21T08:43:34.087310+0000","last_became_peered":"2024-03-21T08:43:34.087310+0000","last_unstale":"2024-03-21T08:43:35.266463+0000","last_undegraded":"2024-03-21T08:43:35.266463+0000","last_fullsized":"2024-03-21T08:43:35.266463+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T16:49:15.990964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063260+0000","last_change":"2024-03-21T08:43:33.081506+0000","last_active":"2024-03-21T08:43:35.063260+0000","last_peered":"2024-03-21T08:43:35.063260+0000","last_clean":"2024-03-21T08:43:35.063260+0000","last_became_active":"2024-03-21T08:43:33.080894+0000","last_became_peered":"2024-03-21T08:43:33.080894+0000","last_unstale":"2024-03-21T08:43:35.063260+0000","last_undegraded":"2024-03-21T08:43:35.063260+0000","last_fullsized":"2024-03-21T08:43:35.063260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:50:21.904297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.064417+0000","last_change":"2024-03-21T08:43:34.089683+0000","last_active":"2024-03-21T08:43:35.064417+0000","last_peered":"2024-03-21T08:43:35.064417+0000","last_clean":"2024-03-21T08:43:35.064417+0000","last_became_active":"2024-03-21T08:43:34.089436+0000","last_became_peered":"2024-03-21T08:43:34.089436+0000","last_unstale":"2024-03-21T08:43:35.064417+0000","last_undegraded":"2024-03-21T08:43:35.064417+0000","last_fullsized":"2024-03-21T08:43:35.064417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:00:59.297657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.265555+0000","last_change":"2024-03-21T08:43:33.080132+0000","last_active":"2024-03-21T08:43:35.265555+0000","last_peered":"2024-03-21T08:43:35.265555+0000","last_clean":"2024-03-21T08:43:35.265555+0000","last_became_active":"2024-03-21T08:43:33.079867+0000","last_became_peered":"2024-03-21T08:43:33.079867+0000","last_unstale":"2024-03-21T08:43:35.265555+0000","last_undegraded":"2024-03-21T08:43:35.265555+0000","last_fullsized":"2024-03-21T08:43:35.265555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T12:14:22.990820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063383+0000","last_change":"2024-03-21T08:43:34.084715+0000","last_active":"2024-03-21T08:43:35.063383+0000","last_peered":"2024-03-21T08:43:35.063383+0000","last_clean":"2024-03-21T08:43:35.063383+0000","last_became_active":"2024-03-21T08:43:34.084436+0000","last_became_peered":"2024-03-21T08:43:34.084436+0000","last_unstale":"2024-03-21T08:43:35.063383+0000","last_undegraded":"2024-03-21T08:43:35.063383+0000","last_fullsized":"2024-03-21T08:43:35.063383+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T14:32:00.940255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063272+0000","last_change":"2024-03-21T08:43:33.076471+0000","last_active":"2024-03-21T08:43:35.063272+0000","last_peered":"2024-03-21T08:43:35.063272+0000","last_clean":"2024-03-21T08:43:35.063272+0000","last_became_active":"2024-03-21T08:43:33.076185+0000","last_became_peered":"2024-03-21T08:43:33.076185+0000","last_unstale":"2024-03-21T08:43:35.063272+0000","last_undegraded":"2024-03-21T08:43:35.063272+0000","last_fullsized":"2024-03-21T08:43:35.063272+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266350+0000","last_change":"2024-03-21T08:43:34.087249+0000","last_active":"2024-03-21T08:43:35.266350+0000","last_peered":"2024-03-21T08:43:35.266350+0000","last_clean":"2024-03-21T08:43:35.266350+0000","last_became_active":"2024-03-21T08:43:34.085986+0000","last_became_peered":"2024-03-21T08:43:34.085986+0000","last_unstale":"2024-03-21T08:43:35.266350+0000","last_undegraded":"2024-03-21T08:43:35.266350+0000","last_fullsized":"2024-03-21T08:43:35.266350+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:31:03.090585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266389+0000","last_change":"2024-03-21T08:43:33.074819+0000","last_active":"2024-03-21T08:43:35.266389+0000","last_peered":"2024-03-21T08:43:35.266389+0000","last_clean":"2024-03-21T08:43:35.266389+0000","last_became_active":"2024-03-21T08:43:33.074585+0000","last_became_peered":"2024-03-21T08:43:33.074585+0000","last_unstale":"2024-03-21T08:43:35.266389+0000","last_undegraded":"2024-03-21T08:43:35.266389+0000","last_fullsized":"2024-03-21T08:43:35.266389+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T19:24:38.393133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.266282+0000","last_change":"2024-03-21T08:43:34.099486+0000","last_active":"2024-03-21T08:43:35.266282+0000","last_peered":"2024-03-21T08:43:35.266282+0000","last_clean":"2024-03-21T08:43:35.266282+0000","last_became_active":"2024-03-21T08:43:34.096768+0000","last_became_peered":"2024-03-21T08:43:34.096768+0000","last_unstale":"2024-03-21T08:43:35.266282+0000","last_undegraded":"2024-03-21T08:43:35.266282+0000","last_fullsized":"2024-03-21T08:43:35.266282+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:33.047253+0000","last_clean_scrub_stamp":"2024-03-21T08:43:33.047253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T09:08:48.617569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.085919+0000","last_change":"2024-03-21T08:43:33.081102+0000","last_active":"2024-03-21T08:43:35.085919+0000","last_peered":"2024-03-21T08:43:35.085919+0000","last_clean":"2024-03-21T08:43:35.085919+0000","last_became_active":"2024-03-21T08:43:33.080820+0000","last_became_peered":"2024-03-21T08:43:33.080820+0000","last_unstale":"2024-03-21T08:43:35.085919+0000","last_undegraded":"2024-03-21T08:43:35.085919+0000","last_fullsized":"2024-03-21T08:43:35.085919+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:43:32.041027+0000","last_clean_scrub_stamp":"2024-03-21T08:43:32.041027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T17:01:22.101232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":231,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-21T08:43:35.063982+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:43:35.063982+0000","last_peered":"2024-03-21T08:43:35.063982+0000","last_clean":"2024-03-21T08:43:35.063982+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:43:35.063982+0000","last_undegraded":"2024-03-21T08:43:35.063982+0000","last_fullsized":"2024-03-21T08:43:35.063982+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509477,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1241088,"data_stored":878429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":638976,"data_stored":287901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509476,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509477,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":643072,"data_stored":287991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":288553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542180,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":289513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542179,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1249280,"data_stored":880463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:43:43.416 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-21T08:43:43.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-21T08:43:43.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 Processor -- start 2024-03-21T08:43:43.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 -- start start 2024-03-21T08:43:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb70166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7015fba0 con 0x7feb7015bd20 2024-03-21T08:43:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7015fe80 con 0x7feb700b0e80 2024-03-21T08:43:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.597+0000 7feb77400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7008bfd0 con 0x7feb700cebd0 2024-03-21T08:43:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb74a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb7015bd20 0x7feb700b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-03-21T08:43:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb74a00640 1 -- 172.21.15.28:0/2493088699 learned_addr learned my addr 172.21.15.28:0/2493088699 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb75400640 1 --2- 172.21.15.28:0/2493088699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb70166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659830505 0 0) 0x7feb7015fba0 con 0x7feb7015bd20 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb54003530 con 0x7feb7015bd20 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb75400640 1 -- 172.21.15.28:0/2493088699 >> v1:172.21.15.151:6789/0 conn(0x7feb700b0e80 legacy=0x7feb700ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578221744 0 0) 0x7feb7015fe80 con 0x7feb700b0e80 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb75400640 1 -- 172.21.15.28:0/2493088699 >> v1:172.21.15.28:6789/0 conn(0x7feb7015bd20 legacy=0x7feb700b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.601+0000 7feb75400640 1 -- 172.21.15.28:0/2493088699 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb70160160 con 0x7feb700cebd0 2024-03-21T08:43:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb75400640 1 --2- 172.21.15.28:0/2493088699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb70166d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb68009dc0 tx=0x7feb6801c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cef84cd0e31fba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680b2070 con 0x7feb700cebd0 2024-03-21T08:43:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb680a5850 con 0x7feb700cebd0 2024-03-21T08:43:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb6fe00640 1 -- 172.21.15.28:0/2493088699 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb680a5c20 con 0x7feb700cebd0 2024-03-21T08:43:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 -- 172.21.15.28:0/2493088699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 msgr2=0x7feb70166d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 --2- 172.21.15.28:0/2493088699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb70166d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb68009dc0 tx=0x7feb6801c990 comp rx=0 tx=0).stop 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 -- 172.21.15.28:0/2493088699 shutdown_connections 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 --2- 172.21.15.28:0/2493088699 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb70166d00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 -- 172.21.15.28:0/2493088699 >> 172.21.15.28:0/2493088699 conn(0x7feb700d1ab0 msgr2=0x7feb700d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 -- 172.21.15.28:0/2493088699 shutdown_connections 2024-03-21T08:43:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.605+0000 7feb77400640 1 -- 172.21.15.28:0/2493088699 wait complete. 2024-03-21T08:43:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 Processor -- start 2024-03-21T08:43:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- start start 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb700c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 0x7feb70166800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb70169be0 con 0x7feb700b0e80 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb70169a60 con 0x7feb7015bd20 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb70169d60 con 0x7feb700cebd0 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50592/0 (socket says 172.21.15.28:50592) 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 -- 172.21.15.28:0/766721823 learned_addr learned my addr 172.21.15.28:0/766721823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb67e00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb700c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb75400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 0x7feb70166800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 -- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 msgr2=0x7feb700c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb700c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 -- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 msgr2=0x7feb70166800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 0x7feb70166800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 -- 172.21.15.28:0/766721823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700c8d20 con 0x7feb700b0e80 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb75400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 0x7feb70166800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb67e00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb700c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb74a00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c3c0 tx=0x7feb6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60017070 con 0x7feb700b0e80 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb60005cf0 con 0x7feb700b0e80 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb600072f0 con 0x7feb700b0e80 2024-03-21T08:43:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb700c9000 con 0x7feb700b0e80 2024-03-21T08:43:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb700c9730 con 0x7feb700b0e80 2024-03-21T08:43:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.609+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7006ffd0 con 0x7feb700b0e80 2024-03-21T08:43:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.613+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7feb6001a070 con 0x7feb700b0e80 2024-03-21T08:43:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.613+0000 7feb6d600640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 0x7feb4c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.613+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb600854d0 con 0x7feb700b0e80 2024-03-21T08:43:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.613+0000 7feb67e00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 0x7feb4c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.613+0000 7feb67e00640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 0x7feb4c069fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7feb701fa140 tx=0x7feb58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.621+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb60055d40 con 0x7feb700b0e80 2024-03-21T08:43:43.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.841+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feb7004f4f0 con 0x7feb700b0e80 2024-03-21T08:43:43.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.841+0000 7feb6d600640 1 -- 172.21.15.28:0/766721823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7feb60055640 con 0x7feb700b0e80 2024-03-21T08:43:43.846 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:43:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:43:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:43:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:43:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-21T08:43:43.847 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:43:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 msgr2=0x7feb4c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 0x7feb4c069fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7feb701fa140 tx=0x7feb58009040 comp rx=0 tx=0).stop 2024-03-21T08:43:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 msgr2=0x7feb70153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7feb6000c3c0 tx=0x7feb6000c880 comp rx=0 tx=0).stop 2024-03-21T08:43:43.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 shutdown_connections 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feb7015bd20 0x7feb70166800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feb4c067b20 0x7feb4c069fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feb700cebd0 0x7feb700c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 --2- 172.21.15.28:0/766721823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb700b0e80 0x7feb70153880 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 >> 172.21.15.28:0/766721823 conn(0x7feb700d1ab0 msgr2=0x7feb700701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 shutdown_connections 2024-03-21T08:43:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:43.845+0000 7feb77400640 1 -- 172.21.15.28:0/766721823 wait complete. 2024-03-21T08:43:43.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:43:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.005+0000 7f0e6b400640 1 Processor -- start 2024-03-21T08:43:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.005+0000 7f0e6b400640 1 -- start start 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.005+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e64099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e6b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e640cdb50 con 0x7f0e640c5f80 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e6b400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e640cde30 con 0x7f0e6415f260 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e6b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6409a520 con 0x7f0e64099b80 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e69400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e640c5f80 0x7f0e640c63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36854/0 (socket says 172.21.15.28:36854) 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e69400640 1 -- 172.21.15.28:0/3973035549 learned_addr learned my addr 172.21.15.28:0/3973035549 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e5be00640 1 --2- 172.21.15.28:0/3973035549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e64099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486190135 0 0) 0x7f0e640cdb50 con 0x7f0e640c5f80 2024-03-21T08:43:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e48003530 con 0x7f0e640c5f80 2024-03-21T08:43:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e5be00640 1 -- 172.21.15.28:0/3973035549 >> v1:172.21.15.151:6789/0 conn(0x7f0e6415f260 legacy=0x7f0e640cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75199819 0 0) 0x7f0e640cde30 con 0x7f0e6415f260 2024-03-21T08:43:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e5be00640 1 -- 172.21.15.28:0/3973035549 >> v1:172.21.15.28:6789/0 conn(0x7f0e640c5f80 legacy=0x7f0e640c63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e5be00640 1 -- 172.21.15.28:0/3973035549 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640c5780 con 0x7f0e64099b80 2024-03-21T08:43:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.009+0000 7f0e5be00640 1 --2- 172.21.15.28:0/3973035549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e64099fe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0097f0 tx=0x7f0e4c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26f0fcba12fce702 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b2070 con 0x7f0e64099b80 2024-03-21T08:43:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0a9070 con 0x7f0e64099b80 2024-03-21T08:43:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e63e00640 1 -- 172.21.15.28:0/3973035549 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0ad880 con 0x7f0e64099b80 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- 172.21.15.28:0/3973035549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 msgr2=0x7f0e64099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3973035549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e64099fe0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0097f0 tx=0x7f0e4c01c450 comp rx=0 tx=0).stop 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- 172.21.15.28:0/3973035549 shutdown_connections 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3973035549 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e64099fe0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- 172.21.15.28:0/3973035549 >> 172.21.15.28:0/3973035549 conn(0x7f0e640b34d0 msgr2=0x7f0e640b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- 172.21.15.28:0/3973035549 shutdown_connections 2024-03-21T08:43:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- 172.21.15.28:0/3973035549 wait complete. 2024-03-21T08:43:44.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 Processor -- start 2024-03-21T08:43:44.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.013+0000 7f0e6b400640 1 -- start start 2024-03-21T08:43:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e6415beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 0x7f0e641f9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6415f7c0 con 0x7f0e640c5f80 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6415f640 con 0x7f0e6415f260 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64169a50 con 0x7f0e64099b80 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50600/0 (socket says 172.21.15.28:50600) 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 -- 172.21.15.28:0/3924847972 learned_addr learned my addr 172.21.15.28:0/3924847972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e68a00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e6415beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e69400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 0x7f0e641f9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 -- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 msgr2=0x7f0e6415beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e6415beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 -- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 msgr2=0x7f0e641f9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 0x7f0e641f9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 -- 172.21.15.28:0/3924847972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640b6710 con 0x7f0e640c5f80 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e69400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 0x7f0e641f9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e5be00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0097f0 tx=0x7f0e4c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b2050 con 0x7f0e640c5f80 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0b3040 con 0x7f0e640c5f80 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e640b69b0 con 0x7f0e640c5f80 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0b9450 con 0x7f0e640c5f80 2024-03-21T08:43:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e640b7210 con 0x7f0e640c5f80 2024-03-21T08:43:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.017+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e640b6d60 con 0x7f0e640c5f80 2024-03-21T08:43:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.021+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0e4c0c0030 con 0x7f0e640c5f80 2024-03-21T08:43:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.021+0000 7f0e61600640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 0x7f0e40069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.021+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0e4c11fbd0 con 0x7f0e640c5f80 2024-03-21T08:43:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.021+0000 7f0e68a00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 0x7f0e40069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.021+0000 7f0e68a00640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 0x7f0e40069fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000f70 tx=0x7f0e54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.025+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0e4c0160d0 con 0x7f0e640c5f80 2024-03-21T08:43:44.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.225+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e640cdcf0 con 0x7f0e640c5f80 2024-03-21T08:43:44.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.225+0000 7f0e61600640 1 -- 172.21.15.28:0/3924847972 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f0e4c0f0440 con 0x7f0e640c5f80 2024-03-21T08:43:44.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:44.229 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:43:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:43:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 msgr2=0x7f0e40069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 0x7f0e40069fe0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000f70 tx=0x7f0e54009040 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 msgr2=0x7f0e6415ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0097f0 tx=0x7f0e4c01ac90 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 shutdown_connections 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0e6415f260 0x7f0e641f9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e640c5f80 0x7f0e6415ce30 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0e40067b20 0x7f0e40069fe0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 --2- 172.21.15.28:0/3924847972 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0e64099b80 0x7f0e6415beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 >> 172.21.15.28:0/3924847972 conn(0x7f0e640b34d0 msgr2=0x7f0e640ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 shutdown_connections 2024-03-21T08:43:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/3924847972 wait complete. 2024-03-21T08:43:44.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-21T08:43:33.457005+0000', 'modified': '2024-03-21T08:43:36.469927+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6249}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6249': {'gid': 6249, 'name': 'a', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/460337969', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 460337969}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 460337969}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6249, 'qdb_cluster': [6249]} 2024-03-21T08:43:44.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-21T08:43:44.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-21T08:43:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.793+0000 7f0500c00640 1 Processor -- start 2024-03-21T08:43:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.793+0000 7f0500c00640 1 -- start start 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f0500c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04fc15bd20 0x7f04fc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f0500c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc0cebd0 0x7f04fc08afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f0500c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc15fba0 con 0x7f04fc0b0e80 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f0500c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc15fe80 con 0x7f04fc15bd20 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f0500c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc160160 con 0x7f04fc0cebd0 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f04fc0b0e80 0x7f04fc0ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36870/0 (socket says 172.21.15.28:36870) 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f1400640 1 -- 172.21.15.28:0/2970607724 learned_addr learned my addr 172.21.15.28:0/2970607724 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2970607724 >> v2:172.21.15.28:3300/0 conn(0x7f04fc15bd20 0x7f04fc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04fa800640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc0cebd0 0x7f04fc08afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 -- 172.21.15.28:0/2970607724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc0cebd0 msgr2=0x7f04fc08afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc0cebd0 0x7f04fc08afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9400640 1 -- 172.21.15.28:0/2970607724 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540663760 0 0) 0x7f04fc15fba0 con 0x7f04fc0b0e80 2024-03-21T08:43:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 -- 172.21.15.28:0/2970607724 >> v1:172.21.15.28:6789/0 conn(0x7f04fc0b0e80 legacy=0x7f04fc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 -- 172.21.15.28:0/2970607724 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc160440 con 0x7f04fc15bd20 2024-03-21T08:43:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.797+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc15bd20 0x7f04fc0b2c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0097c0 tx=0x7f04ec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ecd9f6d202e03e server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9400640 1 -- 172.21.15.28:0/2970607724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0b0070 con 0x7f04fc15bd20 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9400640 1 -- 172.21.15.28:0/2970607724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0ab600 con 0x7f04fc15bd20 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9400640 1 -- 172.21.15.28:0/2970607724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0ab9d0 con 0x7f04fc15bd20 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- 172.21.15.28:0/2970607724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc15bd20 msgr2=0x7f04fc0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc15bd20 0x7f04fc0b2c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0097c0 tx=0x7f04ec01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- 172.21.15.28:0/2970607724 shutdown_connections 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc0cebd0 0x7f04fc08afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- 172.21.15.28:0/2970607724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc15bd20 0x7f04fc0b2c40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- 172.21.15.28:0/2970607724 >> 172.21.15.28:0/2970607724 conn(0x7f04fc0d1ab0 msgr2=0x7f04fc0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- 172.21.15.28:0/2970607724 shutdown_connections 2024-03-21T08:43:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- 172.21.15.28:0/2970607724 wait complete. 2024-03-21T08:43:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 Processor -- start 2024-03-21T08:43:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- start start 2024-03-21T08:43:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04fc0cebd0 0x7f04fc147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 0x7f04fc158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc08bd10 con 0x7f04fc0b0e80 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc15c310 con 0x7f04fc0cebd0 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f0500c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc08be90 con 0x7f04fc15bd20 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50618/0 (socket says 172.21.15.28:50618) 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.801+0000 7f04f9e00640 1 -- 172.21.15.28:0/158901994 learned_addr learned my addr 172.21.15.28:0/158901994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f1400640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04fc0cebd0 0x7f04fc147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04fa800640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 0x7f04fc158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 -- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 msgr2=0x7f04fc158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 0x7f04fc158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 -- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04fc0cebd0 msgr2=0x7f04fc147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04fc0cebd0 0x7f04fc147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 -- 172.21.15.28:0/158901994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc154fd0 con 0x7f04fc0b0e80 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04fa800640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 0x7f04fc158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f9e00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f04ec01a0d0 tx=0x7f04ec019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f2a00640 1 -- 172.21.15.28:0/158901994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0b0040 con 0x7f04fc0b0e80 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f2a00640 1 -- 172.21.15.28:0/158901994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0ba070 con 0x7f04fc0b0e80 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f2a00640 1 -- 172.21.15.28:0/158901994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0b4630 con 0x7f04fc0b0e80 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc1552b0 con 0x7f04fc0b0e80 2024-03-21T08:43:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc06d010 con 0x7f04fc0b0e80 2024-03-21T08:43:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.805+0000 7f04f2a00640 1 -- 172.21.15.28:0/158901994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f04ec0b47d0 con 0x7f04fc0b0e80 2024-03-21T08:43:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f2a00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 0x7f04d8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f2a00640 1 -- 172.21.15.28:0/158901994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04ec11fe20 con 0x7f04fc0b0e80 2024-03-21T08:43:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f1400640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 0x7f04d8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 Processor -- start 2024-03-21T08:43:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- start start 2024-03-21T08:43:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f1400640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 0x7f04d8069fe0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f04fc1fa4c0 tx=0x7f04e4007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca186d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca13cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f8a680 con 0x55e7ca13cff0 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f11b00 con 0x55e7ca187240 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f86720 con 0x55e7ca186920 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50630/0 (socket says 172.21.15.28:50630) 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 -- 172.21.15.28:0/3168107628 learned_addr learned my addr 172.21.15.28:0/3168107628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f9e00640 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca186d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04f1400640 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca13cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 -- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 msgr2=0x55e7ca186d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca186d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 -- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 msgr2=0x55e7ca13cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca13cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 -- 172.21.15.28:0/3168107628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7c9f62940 con 0x55e7ca13cff0 2024-03-21T08:43:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04fa800640 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f04f40074c0 tx=0x7f04f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d467230eacc3f80d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04eaa00640 1 -- 172.21.15.28:0/3168107628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40147e0 con 0x55e7ca13cff0 2024-03-21T08:43:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04eaa00640 1 -- 172.21.15.28:0/3168107628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f401a070 con 0x55e7ca13cff0 2024-03-21T08:43:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f04eaa00640 1 -- 172.21.15.28:0/3168107628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4014ba0 con 0x55e7ca13cff0 2024-03-21T08:43:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- 172.21.15.28:0/3168107628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 msgr2=0x55e7ca13d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f04f40074c0 tx=0x7f04f4009750 comp rx=0 tx=0).stop 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- 172.21.15.28:0/3168107628 shutdown_connections 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13d3d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca13cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 --2- 172.21.15.28:0/3168107628 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca186d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- 172.21.15.28:0/3168107628 >> 172.21.15.28:0/3168107628 conn(0x55e7ca138fa0 msgr2=0x55e7ca13b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- 172.21.15.28:0/3168107628 shutdown_connections 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.809+0000 7f0503dff000 1 -- 172.21.15.28:0/3168107628 wait complete. 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- start start 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 Processor -- start 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca16c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f89990 con 0x55e7ca13cff0 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f44ae0 con 0x55e7ca186920 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f22750 con 0x55e7ca187240 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 10 client.0 _ll_register_callbacks cb 0x55e7ca127ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50634/0 (socket says 172.21.15.28:50634) 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 -- 172.21.15.28:0/3633164232 learned_addr learned my addr 172.21.15.28:0/3633164232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f1400640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca16c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04fa800640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 -- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 msgr2=0x55e7ca16c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 -- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 msgr2=0x55e7ca16c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca16c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 -- 172.21.15.28:0/3633164232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7c9fcfd80 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f9e00640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f04ec000c10 tx=0x7f04ec0ef8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fa533c0af6458 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec11c060 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04ec0f0160 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04f401a070 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04ec0f4050 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 10 client.6402 fetch_fsmap learned FSMap version 28 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e7c9f11d10 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f401a070 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f04ec0b3050 con 0x55e7ca13cff0 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 10 client.6402 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 10 client.6402 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 20 client.6402 populate_metadata read hostname 'smithi028' 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca16d060 0x55e7ca198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 4 client.6402 mds_command: new command op to 6249 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-21T08:43:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 --> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7c9f25b60 con 0x55e7ca16d060 2024-03-21T08:43:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.813+0000 7f04f1400640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca16d060 0x55e7ca198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f04f1400640 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca16d060 0x55e7ca198ea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa7808943318f2d4 server_cookie=779b1d77f2ecaf9e in_seq=0 out_seq=0 2024-03-21T08:43:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f04df400640 10 client.6402 ms_handle_connect on v2:172.21.15.28:6834/460337969 2024-03-21T08:43:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f04df400640 1 -- 172.21.15.28:0/3633164232 <== mds.0 v2:172.21.15.28:6834/460337969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e7c9f25b60 con 0x55e7ca16d060 2024-03-21T08:43:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f04df400640 10 client.6402 handle_command_reply: tid=0 2024-03-21T08:43:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 client.6402 shutdown 2024-03-21T08:43:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 msgr2=0x55e7ca16bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f04ec000c10 tx=0x7f04ec0ef8b0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 shutdown_connections 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca16c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca16d060 0x55e7ca198ea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 --2- 172.21.15.28:0/3633164232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca16bd40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f04df400640 0 client.6402 ms_handle_reset on v2:172.21.15.28:6834/460337969 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 >> 172.21.15.28:0/3633164232 conn(0x55e7ca138fa0 msgr2=0x55e7ca13bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 shutdown_connections 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 1 -- 172.21.15.28:0/3633164232 wait complete. 2024-03-21T08:43:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.817+0000 7f0503dff000 20 client.6402 trim_cache size 0 max 16384 2024-03-21T08:43:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 Processor -- start 2024-03-21T08:43:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 -- start start 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc2033d0 con 0x55e7ca13cff0 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0af520 con 0x55e7ca186920 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0ae920 con 0x55e7ca187240 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50644/0 (socket says 172.21.15.28:50644) 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 -- 172.21.15.28:0/2186450383 learned_addr learned my addr 172.21.15.28:0/2186450383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f1400640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04fa800640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 -- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 msgr2=0x55e7ca16a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 -- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 msgr2=0x55e7ca13c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca13c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f9e00640 1 -- 172.21.15.28:0/2186450383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7c9f57540 con 0x55e7ca13cff0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.833+0000 7f04f1400640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca13c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f04f9e00640 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ec880 tx=0x7f04ec0c0000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91d456efd01b0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f04ea000640 1 -- 172.21.15.28:0/2186450383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0f83f0 con 0x55e7ca13cff0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f04ea000640 1 -- 172.21.15.28:0/2186450383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0f8590 con 0x55e7ca13cff0 2024-03-21T08:43:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f04ea000640 1 -- 172.21.15.28:0/2186450383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec0f0360 con 0x55e7ca13cff0 2024-03-21T08:43:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- 172.21.15.28:0/2186450383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 msgr2=0x55e7ca13c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ec880 tx=0x7f04ec0c0000 comp rx=0 tx=0).stop 2024-03-21T08:43:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- 172.21.15.28:0/2186450383 shutdown_connections 2024-03-21T08:43:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca187240 0x55e7ca16a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca186920 0x55e7ca13c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 --2- 172.21.15.28:0/2186450383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca13c3a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- 172.21.15.28:0/2186450383 >> 172.21.15.28:0/2186450383 conn(0x55e7ca138fa0 msgr2=0x55e7ca1695d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- 172.21.15.28:0/2186450383 shutdown_connections 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- 172.21.15.28:0/2186450383 wait complete. 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 -- start start 2024-03-21T08:43:44.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.837+0000 7f0503dff000 1 Processor -- start 2024-03-21T08:43:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca199920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c9f32a80 con 0x55e7ca13cff0 2024-03-21T08:43:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0aef20 con 0x55e7ca187240 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7ca047700 con 0x55e7ca186920 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 10 client.0 _ll_register_callbacks cb 0x7f04ec0efa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50658/0 (socket says 172.21.15.28:50658) 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 -- 172.21.15.28:0/1141804807 learned_addr learned my addr 172.21.15.28:0/1141804807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f1400640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04fa800640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 -- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 msgr2=0x55e7ca1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 -- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 msgr2=0x55e7ca199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 -- 172.21.15.28:0/1141804807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7c9f1e4b0 con 0x55e7ca13cff0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f9e00640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ee1b0 tx=0x7f04ec0c1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14a36591946f9f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec11c040 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04ec0ef350 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04ec0f8590 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04ec0f83f0 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 10 client.6408 fetch_fsmap learned FSMap version 28 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e7c9f312f0 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0f8590 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f04ec0b3040 con 0x55e7ca13cff0 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 10 client.6408 fetch_fsmap finished waiting for FSMap version 28 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 10 client.6408 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 20 client.6408 populate_metadata read hostname 'smithi028' 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca199e60 0x55e7ca19a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 4 client.6408 mds_command: new command op to 6249 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-21T08:43:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 --> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e7c9f97bc0 con 0x55e7ca199e60 2024-03-21T08:43:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f1400640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca199e60 0x55e7ca19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04f1400640 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca199e60 0x55e7ca19a240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf178c9e69c7263b server_cookie=b73bc2350baa0b76 in_seq=0 out_seq=0 2024-03-21T08:43:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 10 client.6408 ms_handle_connect on v2:172.21.15.28:6834/460337969 2024-03-21T08:43:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 1 -- 172.21.15.28:0/1141804807 <== mds.0 v2:172.21.15.28:6834/460337969 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e7c9f97bc0 con 0x55e7ca199e60 2024-03-21T08:43:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 10 client.6408 handle_command_reply: tid=0 2024-03-21T08:43:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 client.6408 shutdown 2024-03-21T08:43:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 msgr2=0x55e7ca198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ee1b0 tx=0x7f04ec0c1040 comp rx=0 tx=0).stop 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 shutdown_connections 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x55e7ca187240 0x55e7ca199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x55e7ca186920 0x55e7ca1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:6834/460337969,v1:172.21.15.28:6835/460337969] conn(0x55e7ca199e60 0x55e7ca19a240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 --2- 172.21.15.28:0/1141804807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7ca13cff0 0x55e7ca198ea0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 >> 172.21.15.28:0/1141804807 conn(0x55e7ca138fa0 msgr2=0x55e7ca169e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f04df400640 0 client.6408 ms_handle_reset on v2:172.21.15.28:6834/460337969 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 shutdown_connections 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 1 -- 172.21.15.28:0/1141804807 wait complete. 2024-03-21T08:43:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.841+0000 7f0503dff000 20 client.6408 trim_cache size 0 max 16384 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "51dba44a-cba1-4b57-a836-26a407968834", 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6249, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 10.772783681, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 54, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 54, 2024-03-21T08:43:44.851 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 15.652014082000001 2024-03-21T08:43:44.852 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:43:44.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 msgr2=0x7f04d8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 0x7f04d8069fe0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f04fc1fa4c0 tx=0x7f04e4007c10 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 msgr2=0x7f04fc0b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f04ec01a0d0 tx=0x7f04ec019240 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 shutdown_connections 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f04d8067b20 0x7f04d8069fe0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04fc15bd20 0x7f04fc158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04fc0cebd0 0x7f04fc147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 --2- 172.21.15.28:0/158901994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04fc0b0e80 0x7f04fc0b0760 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 >> 172.21.15.28:0/158901994 conn(0x7f04fc0d1ab0 msgr2=0x7f04fc0b3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 shutdown_connections 2024-03-21T08:43:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:44.845+0000 7f0500c00640 1 -- 172.21.15.28:0/158901994 wait complete. 2024-03-21T08:43:44.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:43:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.049+0000 7f546ee00640 1 Processor -- start 2024-03-21T08:43:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.049+0000 7f546ee00640 1 -- start start 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ee00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f546815e0c0 0x7f546806cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5468099b80 0x7f5468099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54680cc1d0 con 0x7f54680ccc70 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ee00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680d1730 con 0x7f546815e0c0 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546809a520 con 0x7f5468099b80 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54680ccc70 0x7f54680cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36878/0 (socket says 172.21.15.28:36878) 2024-03-21T08:43:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f546ce00640 1 -- 172.21.15.28:0/1082696835 learned_addr learned my addr 172.21.15.28:0/1082696835 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f545fe00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5468099b80 0x7f5468099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467e00640 1 --2- 172.21.15.28:0/1082696835 >> v2:172.21.15.151:3300/0 conn(0x7f546815e0c0 0x7f546806cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467400640 1 -- 172.21.15.28:0/1082696835 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995325628 0 0) 0x7f54680cc1d0 con 0x7f54680ccc70 2024-03-21T08:43:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467400640 1 -- 172.21.15.28:0/1082696835 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544c003530 con 0x7f54680ccc70 2024-03-21T08:43:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467e00640 1 -- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5468099b80 msgr2=0x7f5468099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:45.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467e00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5468099b80 0x7f5468099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467e00640 1 -- 172.21.15.28:0/1082696835 >> v1:172.21.15.28:6789/0 conn(0x7f54680ccc70 legacy=0x7f54680cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.053+0000 7f5467e00640 1 -- 172.21.15.28:0/1082696835 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546806d450 con 0x7f546815e0c0 2024-03-21T08:43:45.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f5467e00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f546815e0c0 0x7f546806cf10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54580097c0 tx=0x7f545801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=571455ca13c88a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f5467400640 1 -- 172.21.15.28:0/1082696835 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580a7070 con 0x7f546815e0c0 2024-03-21T08:43:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f5467400640 1 -- 172.21.15.28:0/1082696835 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54580b0070 con 0x7f546815e0c0 2024-03-21T08:43:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f5467400640 1 -- 172.21.15.28:0/1082696835 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580ab890 con 0x7f546815e0c0 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f546815e0c0 msgr2=0x7f546806cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f546815e0c0 0x7f546806cf10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54580097c0 tx=0x7f545801c470 comp rx=0 tx=0).stop 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- 172.21.15.28:0/1082696835 shutdown_connections 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5468099b80 0x7f5468099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- 172.21.15.28:0/1082696835 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f546815e0c0 0x7f546806cf10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- 172.21.15.28:0/1082696835 >> 172.21.15.28:0/1082696835 conn(0x7f54680b34d0 msgr2=0x7f54680b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- 172.21.15.28:0/1082696835 shutdown_connections 2024-03-21T08:43:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- 172.21.15.28:0/1082696835 wait complete. 2024-03-21T08:43:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 Processor -- start 2024-03-21T08:43:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- start start 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 0x7f5468153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 0x7f546815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680d10d0 con 0x7f5468099b80 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546815e4a0 con 0x7f54680ccc70 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.057+0000 7f546ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546816ac90 con 0x7f546815e0c0 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50662/0 (socket says 172.21.15.28:50662) 2024-03-21T08:43:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 -- 172.21.15.28:0/678789082 learned_addr learned my addr 172.21.15.28:0/678789082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f545fe00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 0x7f5468153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 -- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 msgr2=0x7f546815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f546ce00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 0x7f546815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 0x7f546815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 -- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 msgr2=0x7f5468153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 0x7f5468153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 -- 172.21.15.28:0/678789082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680c8270 con 0x7f5468099b80 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f545fe00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 0x7f5468153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f546ce00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 0x7f546815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5467e00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f54580ad040 tx=0x7f545801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580bf610 con 0x7f5468099b80 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54580bf7f0 con 0x7f5468099b80 2024-03-21T08:43:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54680c5f20 con 0x7f5468099b80 2024-03-21T08:43:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580bfbe0 con 0x7f5468099b80 2024-03-21T08:43:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54680c64e0 con 0x7f5468099b80 2024-03-21T08:43:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546804f410 con 0x7f5468099b80 2024-03-21T08:43:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.061+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f54580bd020 con 0x7f5468099b80 2024-03-21T08:43:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.065+0000 7f5464c00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 0x7f544406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.065+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5458120750 con 0x7f5468099b80 2024-03-21T08:43:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.065+0000 7f545fe00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 0x7f544406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.065+0000 7f545fe00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 0x7f544406a010 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5450009a40 tx=0x7f5450008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:45.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.069+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5458016090 con 0x7f5468099b80 2024-03-21T08:43:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.265+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f54680d0aa0 con 0x7f5468099b80 2024-03-21T08:43:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.265+0000 7f5464c00640 1 -- 172.21.15.28:0/678789082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7f54580f10e0 con 0x7f5468099b80 2024-03-21T08:43:45.270 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-21T08:43:45.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 msgr2=0x7f544406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 0x7f544406a010 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5450009a40 tx=0x7f5450008040 comp rx=0 tx=0).stop 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 msgr2=0x7f54680c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f54580ad040 tx=0x7f545801acb0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 shutdown_connections 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5444067b50 0x7f544406a010 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f546815e0c0 0x7f546815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f54680ccc70 0x7f5468153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 --2- 172.21.15.28:0/678789082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5468099b80 0x7f54680c7c00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 >> 172.21.15.28:0/678789082 conn(0x7f54680b34d0 msgr2=0x7f5468070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 shutdown_connections 2024-03-21T08:43:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:45.269+0000 7f546ee00640 1 -- 172.21.15.28:0/678789082 wait complete. 2024-03-21T08:43:45.291 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:43:45.292 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:45.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.293 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:43:45.304 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:43:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42244sec preferred_lft 42244sec 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:43:45.307 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:45.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.308 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:43:45.365 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:43:45.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.368 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:43:45.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.420 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:43:45.420 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:43:45.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:43:45.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:43:45.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:43:45.421 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:43:45.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.512 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:43:45.521 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:43:45.521 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:43:45.522 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:45.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:43:45.582 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:43:45.582 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:45.582 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:45.634 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:45.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:45.635 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:45.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:45.746 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:45.800 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:45.801 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:45.802 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:45.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.803 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:45.854 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:43:45.855 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:43:45.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:45.926 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:45.982 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:45.983 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:45.984 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:45.985 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:45.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:45.986 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:45.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:43:45.997+0000 7f6b9f1d7580 -1 init, newargv = 0x5584cff833c0 newargc=15 2024-03-21T08:43:46.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40323]: starting ceph client 2024-03-21T08:43:46.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40323]: starting fuse 2024-03-21T08:43:46.011 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:43:46.011 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:43:46.011 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:43:46.012 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:43:46.013 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:46.013 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:43:46.013 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:43:46.143 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.40323.asok 2024-03-21T08:43:46.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:43:46.155 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40323.asok 2024-03-21T08:43:46.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:46.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40323.asok status 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "40323", 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:43:46.332 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6423, 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6423 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 211703040 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:46.333 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 211703040 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.6423 192.168.0.1:0/211703040", 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/211703040", 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 28, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 54, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:43:46.334 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:43:46.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:46.347 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:43:46.347 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:46.356 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:43:46.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:46.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:46.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:46.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:46.422 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-21T08:43:46.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-21T08:43:46.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.673+0000 7f4dfa200640 1 Processor -- start 2024-03-21T08:43:46.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.673+0000 7f4dfa200640 1 -- start start 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df40cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 0x7f4df408afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df41626d0 0x7f4df40ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df406c280 con 0x7f4df41626d0 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df408b8f0 con 0x7f4df415e120 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4dfa200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df408bbd0 con 0x7f4df415dd40 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df41626d0 0x7f4df40ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df41626d0 0x7f4df40ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50676/0 (socket says 172.21.15.28:50676) 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 -- 172.21.15.28:0/2695891767 learned_addr learned my addr 172.21.15.28:0/2695891767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4debe00640 1 --2- 172.21.15.28:0/2695891767 >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 0x7f4df408afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3400640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df40cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 -- 172.21.15.28:0/2695891767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 msgr2=0x7f4df40cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df40cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 -- 172.21.15.28:0/2695891767 >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 msgr2=0x7f4df408afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 --2- 172.21.15.28:0/2695891767 >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 0x7f4df408afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3e00640 1 -- 172.21.15.28:0/2695891767 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df408beb0 con 0x7f4df41626d0 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4df3400640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df40cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.677+0000 7f4debe00640 1 --2- 172.21.15.28:0/2695891767 >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 0x7f4df408afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4df3e00640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df40ce3a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009e50 tx=0x7f4dec01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83506d3c1d4ed5d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4df2a00640 1 -- 172.21.15.28:0/2695891767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0a55e0 con 0x7f4df41626d0 2024-03-21T08:43:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4df2a00640 1 -- 172.21.15.28:0/2695891767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec0a5780 con 0x7f4df41626d0 2024-03-21T08:43:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4df2a00640 1 -- 172.21.15.28:0/2695891767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0a5b50 con 0x7f4df41626d0 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- 172.21.15.28:0/2695891767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 msgr2=0x7f4df40ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df40ce3a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009e50 tx=0x7f4dec01c960 comp rx=0 tx=0).stop 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- 172.21.15.28:0/2695891767 shutdown_connections 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df40ce3a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 --2- 172.21.15.28:0/2695891767 >> v2:172.21.15.151:3300/0 conn(0x7f4df415e120 0x7f4df408afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 --2- 172.21.15.28:0/2695891767 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df40cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- 172.21.15.28:0/2695891767 >> 172.21.15.28:0/2695891767 conn(0x7f4df40d1b20 msgr2=0x7f4df40d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- 172.21.15.28:0/2695891767 shutdown_connections 2024-03-21T08:43:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- 172.21.15.28:0/2695891767 wait complete. 2024-03-21T08:43:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 Processor -- start 2024-03-21T08:43:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.681+0000 7f4dfa200640 1 -- start start 2024-03-21T08:43:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df41519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4df415e120 0x7f4df40b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df406d590 con 0x7f4df41626d0 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df406d410 con 0x7f4df415e120 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df40b1c60 con 0x7f4df415dd40 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50682/0 (socket says 172.21.15.28:50682) 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 -- 172.21.15.28:0/1874522626 learned_addr learned my addr 172.21.15.28:0/1874522626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3400640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df41519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4debe00640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4df415e120 0x7f4df40b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 -- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 msgr2=0x7f4df41519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df41519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 -- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4df415e120 msgr2=0x7f4df40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4df415e120 0x7f4df40b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 -- 172.21.15.28:0/1874522626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df41523b0 con 0x7f4df41626d0 2024-03-21T08:43:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4df3e00640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4dec01b010 tx=0x7f4dec019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0bc020 con 0x7f4df41626d0 2024-03-21T08:43:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec0b8670 con 0x7f4df41626d0 2024-03-21T08:43:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec009510 con 0x7f4df41626d0 2024-03-21T08:43:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df4150ef0 con 0x7f4df41626d0 2024-03-21T08:43:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df41514b0 con 0x7f4df41626d0 2024-03-21T08:43:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df4150d10 con 0x7f4df41626d0 2024-03-21T08:43:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.685+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4dec0bf020 con 0x7f4df41626d0 2024-03-21T08:43:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.689+0000 7f4deb400640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 0x7f4dd006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.689+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4dec120ec0 con 0x7f4df41626d0 2024-03-21T08:43:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.689+0000 7f4df3400640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 0x7f4dd006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:46.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.689+0000 7f4df3400640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 0x7f4dd006a010 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4de40098d0 tx=0x7f4de4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.693+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4df4150d10 con 0x7f4df41626d0 2024-03-21T08:43:46.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.913+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) v1 -- 0x7f4df409fb90 con 0x7f4df41626d0 2024-03-21T08:43:46.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4deb400640 1 -- 172.21.15.28:0/1874522626 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v16) v1 ==== 183+0+140 (secure 0 0 0) 0x7f4dec0c6030 con 0x7f4df41626d0 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-21T08:43:46.920 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 msgr2=0x7f4dd006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 0x7f4dd006a010 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4de40098d0 tx=0x7f4de4008040 comp rx=0 tx=0).stop 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 msgr2=0x7f4df414f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4dec01b010 tx=0x7f4dec019700 comp rx=0 tx=0).stop 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 shutdown_connections 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df41626d0 0x7f4df414f660 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4df415e120 0x7f4df40b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4dd0067b50 0x7f4dd006a010 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 --2- 172.21.15.28:0/1874522626 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4df415dd40 0x7f4df41519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 >> 172.21.15.28:0/1874522626 conn(0x7f4df40d1b20 msgr2=0x7f4df4157780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 shutdown_connections 2024-03-21T08:43:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:46.917+0000 7f4dfa200640 1 -- 172.21.15.28:0/1874522626 wait complete. 2024-03-21T08:43:46.941 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:46.941 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:46.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 Processor -- start 2024-03-21T08:43:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 -- start start 2024-03-21T08:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f738c099b80 0x7f738c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f738c0d0880 0x7f738c099f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c09a4a0 con 0x7f738c0b0b40 2024-03-21T08:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c06d890 con 0x7f738c099b80 2024-03-21T08:43:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7393200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c0cfac0 con 0x7f738c0d0880 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f738be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f738c099b80 0x7f738c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7391200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f738c0b0b40 0x7f738c161d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36890/0 (socket says 172.21.15.28:36890) 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7391200640 1 -- 172.21.15.28:0/1127476678 learned_addr learned my addr 172.21.15.28:0/1127476678 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f738be00640 1 --2- 172.21.15.28:0/1127476678 >> v2:172.21.15.28:3300/0 conn(0x7f738c099b80 0x7f738c15f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50686/0 (socket says 172.21.15.28:50686) 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.157+0000 7f7383400640 1 --2- 172.21.15.28:0/1127476678 >> v2:172.21.15.151:3300/0 conn(0x7f738c0d0880 0x7f738c099f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738be00640 1 -- 172.21.15.28:0/1127476678 >> v2:172.21.15.151:3300/0 conn(0x7f738c0d0880 msgr2=0x7f738c099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738be00640 1 --2- 172.21.15.28:0/1127476678 >> v2:172.21.15.151:3300/0 conn(0x7f738c0d0880 0x7f738c099f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738be00640 1 -- 172.21.15.28:0/1127476678 >> v1:172.21.15.28:6789/0 conn(0x7f738c0b0b40 legacy=0x7f738c161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738be00640 1 -- 172.21.15.28:0/1127476678 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0cfda0 con 0x7f738c099b80 2024-03-21T08:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738be00640 1 --2- 172.21.15.28:0/1127476678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c099b80 0x7f738c15f220 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f738c072df0 tx=0x7f737c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6d7a69514cd3a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738b400640 1 -- 172.21.15.28:0/1127476678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c0b2070 con 0x7f738c099b80 2024-03-21T08:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738b400640 1 -- 172.21.15.28:0/1127476678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c0a5900 con 0x7f738c099b80 2024-03-21T08:43:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f738b400640 1 -- 172.21.15.28:0/1127476678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c0a5cd0 con 0x7f738c099b80 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f7393200640 1 -- 172.21.15.28:0/1127476678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c099b80 msgr2=0x7f738c15f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.161+0000 7f7393200640 1 --2- 172.21.15.28:0/1127476678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c099b80 0x7f738c15f220 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f738c072df0 tx=0x7f737c01c900 comp rx=0 tx=0).stop 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- 172.21.15.28:0/1127476678 shutdown_connections 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 --2- 172.21.15.28:0/1127476678 >> v2:172.21.15.151:3300/0 conn(0x7f738c0d0880 0x7f738c099f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 --2- 172.21.15.28:0/1127476678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c099b80 0x7f738c15f220 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- 172.21.15.28:0/1127476678 >> 172.21.15.28:0/1127476678 conn(0x7f738c0b3510 msgr2=0x7f738c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- 172.21.15.28:0/1127476678 shutdown_connections 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- 172.21.15.28:0/1127476678 wait complete. 2024-03-21T08:43:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 Processor -- start 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- start start 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 0x7f738c1564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f738c0b0b40 0x7f738c146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0b2cb0 con 0x7f738c0d0880 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0b2b30 con 0x7f738c099b80 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7393200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c16b050 con 0x7f738c0b0b40 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50690/0 (socket says 172.21.15.28:50690) 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 -- 172.21.15.28:0/2225577078 learned_addr learned my addr 172.21.15.28:0/2225577078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f738be00640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 0x7f738c1564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7383400640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f738c0b0b40 0x7f738c146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 -- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f738c0b0b40 msgr2=0x7f738c146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f738c0b0b40 0x7f738c146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 -- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 msgr2=0x7f738c1564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 0x7f738c1564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 -- 172.21.15.28:0/2225577078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c15ac00 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f738be00640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 0x7f738c1564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.165+0000 7f7391200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f738400c3c0 tx=0x7f738400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384017070 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384005cf0 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c15b0f0 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73840072f0 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c15b6b0 con 0x7f738c0d0880 2024-03-21T08:43:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c04f410 con 0x7f738c0d0880 2024-03-21T08:43:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f738401a070 con 0x7f738c0d0880 2024-03-21T08:43:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 0x7f7368069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7384085400 con 0x7f738c0d0880 2024-03-21T08:43:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.169+0000 7f738be00640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 0x7f7368069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.173+0000 7f738be00640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 0x7f7368069fe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f737c0af000 tx=0x7f737c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.173+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7384055c80 con 0x7f738c0d0880 2024-03-21T08:43:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.409+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738c0b43f0 con 0x7f738c0d0880 2024-03-21T08:43:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.409+0000 7f7388c00640 1 -- 172.21.15.28:0/2225577078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f7384055580 con 0x7f738c0d0880 2024-03-21T08:43:47.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:47.413 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 msgr2=0x7f7368069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 0x7f7368069fe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f737c0af000 tx=0x7f737c0a7070 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 msgr2=0x7f738c15a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f738400c3c0 tx=0x7f738400c880 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 shutdown_connections 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f738c0d0880 0x7f738c15a6c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7368067b20 0x7f7368069fe0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f738c0b0b40 0x7f738c146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 --2- 172.21.15.28:0/2225577078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f738c099b80 0x7f738c1564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 >> 172.21.15.28:0/2225577078 conn(0x7f738c0b3510 msgr2=0x7f738c09aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 shutdown_connections 2024-03-21T08:43:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.413+0000 7f7393200640 1 -- 172.21.15.28:0/2225577078 wait complete. 2024-03-21T08:43:47.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:47.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 Processor -- start 2024-03-21T08:43:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 -- start start 2024-03-21T08:43:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 0x7f59040cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 0x7f590409a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040ccbd0 con 0x7f590415bdc0 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040cd7e0 con 0x7f5904099c90 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.585+0000 7f590ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040cceb0 con 0x7f59040cc640 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5903e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f590415bdc0 0x7f59040d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36900/0 (socket says 172.21.15.28:36900) 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5903e00640 1 -- 172.21.15.28:0/1225023191 learned_addr learned my addr 172.21.15.28:0/1225023191 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5908e00640 1 --2- 172.21.15.28:0/1225023191 >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 0x7f590409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 --2- 172.21.15.28:0/1225023191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 0x7f59040cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5903400640 1 -- 172.21.15.28:0/1225023191 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078626301 0 0) 0x7f59040ccbd0 con 0x7f590415bdc0 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5903400640 1 -- 172.21.15.28:0/1225023191 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e8003530 con 0x7f590415bdc0 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 -- 172.21.15.28:0/1225023191 >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 msgr2=0x7f590409a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 --2- 172.21.15.28:0/1225023191 >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 0x7f590409a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 -- 172.21.15.28:0/1225023191 >> v1:172.21.15.28:6789/0 conn(0x7f590415bdc0 legacy=0x7f59040d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 -- 172.21.15.28:0/1225023191 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040cd190 con 0x7f59040cc640 2024-03-21T08:43:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f5908e00640 1 --2- 172.21.15.28:0/1225023191 >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 0x7f590409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.589+0000 7f58fbe00640 1 --2- 172.21.15.28:0/1225023191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 0x7f59040cdc70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0097c0 tx=0x7f58ec01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e063bdaa66e68971 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5903400640 1 -- 172.21.15.28:0/1225023191 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0b0070 con 0x7f59040cc640 2024-03-21T08:43:47.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5903400640 1 -- 172.21.15.28:0/1225023191 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec0ab600 con 0x7f59040cc640 2024-03-21T08:43:47.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5903400640 1 -- 172.21.15.28:0/1225023191 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0ab9d0 con 0x7f59040cc640 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- 172.21.15.28:0/1225023191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 msgr2=0x7f59040cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- 172.21.15.28:0/1225023191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 0x7f59040cdc70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0097c0 tx=0x7f58ec01c410 comp rx=0 tx=0).stop 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- 172.21.15.28:0/1225023191 shutdown_connections 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- 172.21.15.28:0/1225023191 >> v2:172.21.15.151:3300/0 conn(0x7f5904099c90 0x7f590409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- 172.21.15.28:0/1225023191 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f59040cc640 0x7f59040cdc70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- 172.21.15.28:0/1225023191 >> 172.21.15.28:0/1225023191 conn(0x7f59040b3620 msgr2=0x7f59040b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- 172.21.15.28:0/1225023191 shutdown_connections 2024-03-21T08:43:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- 172.21.15.28:0/1225023191 wait complete. 2024-03-21T08:43:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 Processor -- start 2024-03-21T08:43:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- start start 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5904099c90 0x7f590415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59040cc640 0x7f590414d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b1cc0 con 0x7f590415bdc0 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b1b40 con 0x7f59040cc640 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f590ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b1e40 con 0x7f5904099c90 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5908e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5908e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50694/0 (socket says 172.21.15.28:50694) 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5908e00640 1 -- 172.21.15.28:0/855365929 learned_addr learned my addr 172.21.15.28:0/855365929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f5903e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5904099c90 0x7f590415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.593+0000 7f58fbe00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59040cc640 0x7f590414d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 -- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5904099c90 msgr2=0x7f590415eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5904099c90 0x7f590415eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 -- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59040cc640 msgr2=0x7f590414d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59040cc640 0x7f590414d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 -- 172.21.15.28:0/855365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040c8c00 con 0x7f590415bdc0 2024-03-21T08:43:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5908e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00efc0 tx=0x7f58fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc018030 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc00cdf0 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590415e160 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc003bc0 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590415e720 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590409faf0 con 0x7f590415bdc0 2024-03-21T08:43:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f58fc01b070 con 0x7f590415bdc0 2024-03-21T08:43:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 0x7f58e0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.597+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f58fc086900 con 0x7f590415bdc0 2024-03-21T08:43:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.601+0000 7f5903e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 0x7f58e0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.601+0000 7f5903e00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 0x7f58e0069fe0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f58f4007fd0 tx=0x7f58f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.605+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f58fc022070 con 0x7f590415bdc0 2024-03-21T08:43:47.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.765+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f590404f4f0 con 0x7f590415bdc0 2024-03-21T08:43:47.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.765+0000 7f5900c00640 1 -- 172.21.15.28:0/855365929 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58fc010040 con 0x7f590415bdc0 2024-03-21T08:43:47.770 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 msgr2=0x7f58e0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 0x7f58e0069fe0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f58f4007fd0 tx=0x7f58f4009040 comp rx=0 tx=0).stop 2024-03-21T08:43:47.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 msgr2=0x7f59040cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00efc0 tx=0x7f58fc00c490 comp rx=0 tx=0).stop 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 shutdown_connections 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590415bdc0 0x7f59040cfe80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f59040cc640 0x7f590414d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f58e0067b20 0x7f58e0069fe0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 --2- 172.21.15.28:0/855365929 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5904099c90 0x7f590415eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 >> 172.21.15.28:0/855365929 conn(0x7f59040b3620 msgr2=0x7f590406d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 shutdown_connections 2024-03-21T08:43:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.769+0000 7f590ae00640 1 -- 172.21.15.28:0/855365929 wait complete. 2024-03-21T08:43:47.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 Processor -- start 2024-03-21T08:43:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 -- start start 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 0x7fd160071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd1600cd300 0x7fd160071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16015bf90 con 0x7fd1600b4800 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16015b1d0 con 0x7fd1600cd300 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd165000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16015b4b0 con 0x7fd1600b4420 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1600b4800 0x7fd1600ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36910/0 (socket says 172.21.15.28:36910) 2024-03-21T08:43:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15d400640 1 -- 172.21.15.28:0/2695499898 learned_addr learned my addr 172.21.15.28:0/2695499898 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15de00640 1 --2- 172.21.15.28:0/2695499898 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 0x7fd160071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15e800640 1 --2- 172.21.15.28:0/2695499898 >> v2:172.21.15.151:3300/0 conn(0x7fd1600cd300 0x7fd160071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557663739 0 0) 0x7fd16015bf90 con 0x7fd1600b4800 2024-03-21T08:43:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd144005740 con 0x7fd1600b4800 2024-03-21T08:43:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086366276 0 0) 0x7fd144005740 con 0x7fd1600b4800 2024-03-21T08:43:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15de00640 1 -- 172.21.15.28:0/2695499898 >> v2:172.21.15.151:3300/0 conn(0x7fd1600cd300 msgr2=0x7fd160071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15de00640 1 --2- 172.21.15.28:0/2695499898 >> v2:172.21.15.151:3300/0 conn(0x7fd1600cd300 0x7fd160071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15de00640 1 -- 172.21.15.28:0/2695499898 >> v1:172.21.15.28:6789/0 conn(0x7fd1600b4800 legacy=0x7fd1600ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.933+0000 7fd15de00640 1 -- 172.21.15.28:0/2695499898 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16015b790 con 0x7fd1600b4420 2024-03-21T08:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd15de00640 1 --2- 172.21.15.28:0/2695499898 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 0x7fd160071a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd1500097c0 tx=0x7fd15001c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66758bb9447bbfd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1500b0070 con 0x7fd1600b4420 2024-03-21T08:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1500ab400 con 0x7fd1600b4420 2024-03-21T08:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd15ca00640 1 -- 172.21.15.28:0/2695499898 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c001680 con 0x7fd1600b4420 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 -- 172.21.15.28:0/2695499898 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 msgr2=0x7fd160071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 --2- 172.21.15.28:0/2695499898 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 0x7fd160071a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd1500097c0 tx=0x7fd15001c300 comp rx=0 tx=0).stop 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 -- 172.21.15.28:0/2695499898 shutdown_connections 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 --2- 172.21.15.28:0/2695499898 >> v2:172.21.15.151:3300/0 conn(0x7fd1600cd300 0x7fd160071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 --2- 172.21.15.28:0/2695499898 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1600b4420 0x7fd160071a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 -- 172.21.15.28:0/2695499898 >> 172.21.15.28:0/2695499898 conn(0x7fd1600674f0 msgr2=0x7fd16006b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 -- 172.21.15.28:0/2695499898 shutdown_connections 2024-03-21T08:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.937+0000 7fd165000640 1 -- 172.21.15.28:0/2695499898 wait complete. 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 Processor -- start 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- start start 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 0x7fd16013d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd16013dea0 0x7fd160154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006e680 con 0x7fd1600b4420 2024-03-21T08:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006e500 con 0x7fd1600cd300 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006e800 con 0x7fd16013dea0 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50700/0 (socket says 172.21.15.28:50700) 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 -- 172.21.15.28:0/3469757933 learned_addr learned my addr 172.21.15.28:0/3469757933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15d400640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 0x7fd16013d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 -- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd16013dea0 msgr2=0x7fd160154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd16013dea0 0x7fd160154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 -- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 msgr2=0x7fd16013d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 0x7fd16013d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 -- 172.21.15.28:0/3469757933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16015e1b0 con 0x7fd1600b4420 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15d400640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 0x7fd16013d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd15de00640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd1500ae040 tx=0x7fd15001ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1500b0050 con 0x7fd1600b4420 2024-03-21T08:43:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1500bc070 con 0x7fd1600b4420 2024-03-21T08:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1500b78b0 con 0x7fd1600b4420 2024-03-21T08:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16013e8f0 con 0x7fd1600b4420 2024-03-21T08:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16013e610 con 0x7fd1600b4420 2024-03-21T08:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.941+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd160073430 con 0x7fd1600b4420 2024-03-21T08:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.945+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd1500b7a50 con 0x7fd1600b4420 2024-03-21T08:43:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.945+0000 7fd156000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 0x7fd14006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.945+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd150120190 con 0x7fd1600b4420 2024-03-21T08:43:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.945+0000 7fd15d400640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 0x7fd14006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.945+0000 7fd15d400640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 0x7fd14006a010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0011f0 tx=0x7fd14c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:47.949+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd1500c5020 con 0x7fd1600b4420 2024-03-21T08:43:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd16004f4f0 con 0x7fd1600b4420 2024-03-21T08:43:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd156000640 1 -- 172.21.15.28:0/3469757933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd150015fb0 con 0x7fd1600b4420 2024-03-21T08:43:48.116 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 msgr2=0x7fd14006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 0x7fd14006a010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0011f0 tx=0x7fd14c009040 comp rx=0 tx=0).stop 2024-03-21T08:43:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 msgr2=0x7fd16013b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd1500ae040 tx=0x7fd15001ab40 comp rx=0 tx=0).stop 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 shutdown_connections 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd140067b50 0x7fd14006a010 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd16013dea0 0x7fd160154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd1600cd300 0x7fd16013d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 --2- 172.21.15.28:0/3469757933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600b4420 0x7fd16013b080 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 >> 172.21.15.28:0/3469757933 conn(0x7fd1600674f0 msgr2=0x7fd16015daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 shutdown_connections 2024-03-21T08:43:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.113+0000 7fd165000640 1 -- 172.21.15.28:0/3469757933 wait complete. 2024-03-21T08:43:48.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 Processor -- start 2024-03-21T08:43:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 -- start start 2024-03-21T08:43:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ca815bd20 0x7f6ca80b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6ca80cea00 0x7f6ca80b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.277+0000 7f6cae600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80cee50 con 0x7f6ca815bd20 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6cae600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca815fba0 con 0x7f6ca80cea00 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6cae600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815fe80 con 0x7f6ca80b0e80 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ca815bd20 0x7f6ca80b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ca815bd20 0x7f6ca80b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50712/0 (socket says 172.21.15.28:50712) 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 -- 172.21.15.28:0/3795210616 learned_addr learned my addr 172.21.15.28:0/3795210616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7e00640 1 --2- 172.21.15.28:0/3795210616 >> v2:172.21.15.151:3300/0 conn(0x7f6ca80cea00 0x7f6ca80b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6c9ea00640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 -- 172.21.15.28:0/3795210616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 msgr2=0x7f6ca80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 -- 172.21.15.28:0/3795210616 >> v2:172.21.15.151:3300/0 conn(0x7f6ca80cea00 msgr2=0x7f6ca80b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 --2- 172.21.15.28:0/3795210616 >> v2:172.21.15.151:3300/0 conn(0x7f6ca80cea00 0x7f6ca80b19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 -- 172.21.15.28:0/3795210616 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca8160160 con 0x7f6ca815bd20 2024-03-21T08:43:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6c9ea00640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca7400640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca815bd20 0x7f6ca80b2c40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80d2b50 tx=0x7f6c9801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4614bb8b5df7a5a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca6a00640 1 -- 172.21.15.28:0/3795210616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980b0070 con 0x7f6ca815bd20 2024-03-21T08:43:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca6a00640 1 -- 172.21.15.28:0/3795210616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c980ab600 con 0x7f6ca815bd20 2024-03-21T08:43:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.281+0000 7f6ca6a00640 1 -- 172.21.15.28:0/3795210616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980ab9d0 con 0x7f6ca815bd20 2024-03-21T08:43:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- 172.21.15.28:0/3795210616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca815bd20 msgr2=0x7f6ca80b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca815bd20 0x7f6ca80b2c40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80d2b50 tx=0x7f6c9801c260 comp rx=0 tx=0).stop 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- 172.21.15.28:0/3795210616 shutdown_connections 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- 172.21.15.28:0/3795210616 >> v2:172.21.15.151:3300/0 conn(0x7f6ca80cea00 0x7f6ca80b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca80ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- 172.21.15.28:0/3795210616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca815bd20 0x7f6ca80b2c40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- 172.21.15.28:0/3795210616 >> 172.21.15.28:0/3795210616 conn(0x7f6ca80d1ab0 msgr2=0x7f6ca80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- 172.21.15.28:0/3795210616 shutdown_connections 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- 172.21.15.28:0/3795210616 wait complete. 2024-03-21T08:43:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 Processor -- start 2024-03-21T08:43:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- start start 2024-03-21T08:43:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ca815bd20 0x7f6ca8070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815c280 con 0x7f6ca80cea00 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca815c100 con 0x7f6ca815bd20 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6cae600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca808afc0 con 0x7f6ca80b0e80 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6c9ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6c9ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50714/0 (socket says 172.21.15.28:50714) 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6c9ea00640 1 -- 172.21.15.28:0/501279941 learned_addr learned my addr 172.21.15.28:0/501279941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6ca7400640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.285+0000 7f6ca7e00640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ca815bd20 0x7f6ca8070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 -- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 msgr2=0x7f6ca814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 -- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ca815bd20 msgr2=0x7f6ca8070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ca815bd20 0x7f6ca8070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 -- 172.21.15.28:0/501279941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca814f460 con 0x7f6ca80cea00 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9ea00640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c7c0 tx=0x7f6c9000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90015070 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90007a30 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90007e00 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca814f740 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca814fd00 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca804f4f0 con 0x7f6ca80cea00 2024-03-21T08:43:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6c90018070 con 0x7f6ca80cea00 2024-03-21T08:43:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 0x7f6c8406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c90085ed0 con 0x7f6ca80cea00 2024-03-21T08:43:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.289+0000 7f6ca7400640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 0x7f6c8406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.293+0000 7f6ca7400640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 0x7f6c8406a010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6c980ad000 tx=0x7f6c980a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.301+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6c900560f0 con 0x7f6ca80cea00 2024-03-21T08:43:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.461+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ca80d2830 con 0x7f6ca80cea00 2024-03-21T08:43:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.461+0000 7f6c9fe00640 1 -- 172.21.15.28:0/501279941 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c9005a190 con 0x7f6ca80cea00 2024-03-21T08:43:48.466 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 msgr2=0x7f6c8406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 0x7f6c8406a010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6c980ad000 tx=0x7f6c980a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 msgr2=0x7f6ca815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c7c0 tx=0x7f6c9000cc80 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 shutdown_connections 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6ca815bd20 0x7f6ca8070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ca80cea00 0x7f6ca815ae00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6c84067b50 0x7f6c8406a010 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 --2- 172.21.15.28:0/501279941 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6ca80b0e80 0x7f6ca814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 >> 172.21.15.28:0/501279941 conn(0x7f6ca80d1ab0 msgr2=0x7f6ca808b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 shutdown_connections 2024-03-21T08:43:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.465+0000 7f6cae600640 1 -- 172.21.15.28:0/501279941 wait complete. 2024-03-21T08:43:48.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 Processor -- start 2024-03-21T08:43:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 -- start start 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ab009a3a0 0x7f5ab00cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00ccbd0 con 0x7f5ab00cc640 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00cceb0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5ab6400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00cd7e0 con 0x7f5ab009a3a0 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5aafe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ab009a3a0 0x7f5ab00cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5aafe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ab009a3a0 0x7f5ab00cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50718/0 (socket says 172.21.15.28:50718) 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5aa6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ab00cc640 0x7f5ab0099c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36918/0 (socket says 172.21.15.28:36918) 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5aafe00640 1 -- 172.21.15.28:0/915979797 learned_addr learned my addr 172.21.15.28:0/915979797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.625+0000 7f5aafe00640 1 -- 172.21.15.28:0/915979797 >> v1:172.21.15.151:6789/0 conn(0x7f5ab015bdc0 legacy=0x7f5ab00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aafe00640 1 -- 172.21.15.28:0/915979797 >> v1:172.21.15.28:6789/0 conn(0x7f5ab00cc640 legacy=0x7f5ab0099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aafe00640 1 -- 172.21.15.28:0/915979797 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00cd190 con 0x7f5ab009a3a0 2024-03-21T08:43:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aaea00640 1 -- 172.21.15.28:0/915979797 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589113904 0 0) 0x7f5ab00ccbd0 con 0x7f5ab00cc640 2024-03-21T08:43:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aafe00640 1 --2- 172.21.15.28:0/915979797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab009a3a0 0x7f5ab00cdc70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009870 tx=0x7f5aa801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e270f47d453115 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aaea00640 1 -- 172.21.15.28:0/915979797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80aa600 con 0x7f5ab009a3a0 2024-03-21T08:43:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aaea00640 1 -- 172.21.15.28:0/915979797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa80af070 con 0x7f5ab009a3a0 2024-03-21T08:43:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5aaea00640 1 -- 172.21.15.28:0/915979797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80aa9c0 con 0x7f5ab009a3a0 2024-03-21T08:43:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 -- 172.21.15.28:0/915979797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab009a3a0 msgr2=0x7f5ab00cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 --2- 172.21.15.28:0/915979797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab009a3a0 0x7f5ab00cdc70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009870 tx=0x7f5aa801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 -- 172.21.15.28:0/915979797 shutdown_connections 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 --2- 172.21.15.28:0/915979797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab009a3a0 0x7f5ab00cdc70 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 -- 172.21.15.28:0/915979797 >> 172.21.15.28:0/915979797 conn(0x7f5ab00b3620 msgr2=0x7f5ab00b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 -- 172.21.15.28:0/915979797 shutdown_connections 2024-03-21T08:43:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.629+0000 7f5ab6400640 1 -- 172.21.15.28:0/915979797 wait complete. 2024-03-21T08:43:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 Processor -- start 2024-03-21T08:43:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 -- start start 2024-03-21T08:43:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab009a3a0 0x7f5ab01fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab00cc640 0x7f5ab0158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00b1cc0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00b1b40 con 0x7f5ab00cc640 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5ab6400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00b1e40 con 0x7f5ab009a3a0 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50734/0 (socket says 172.21.15.28:50734) 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 -- 172.21.15.28:0/2416167109 learned_addr learned my addr 172.21.15.28:0/2416167109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aa6a00640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab00cc640 0x7f5ab0158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aaf400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab009a3a0 0x7f5ab01fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 -- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab009a3a0 msgr2=0x7f5ab01fafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab009a3a0 0x7f5ab01fafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 -- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab00cc640 msgr2=0x7f5ab0158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab00cc640 0x7f5ab0158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 -- 172.21.15.28:0/2416167109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab01fb4f0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.633+0000 7f5aafe00640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80ac040 tx=0x7f5aa801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80bd5f0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa80bd7b0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80bdba0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab00b4960 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab00b4f20 con 0x7f5ab015bdc0 2024-03-21T08:43:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab00b46d0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5aa80a9070 con 0x7f5ab015bdc0 2024-03-21T08:43:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5aa81207e0 con 0x7f5ab015bdc0 2024-03-21T08:43:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aaf400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.637+0000 7f5aaf400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0007fd0 tx=0x7f5aa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.641+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5aa8016150 con 0x7f5ab015bdc0 2024-03-21T08:43:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5ab0166f60 con 0x7f5ab015bdc0 2024-03-21T08:43:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5aa7e00640 1 -- 172.21.15.28:0/2416167109 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5aa80f0970 con 0x7f5ab015bdc0 2024-03-21T08:43:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 msgr2=0x7f5a8c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0007fd0 tx=0x7f5aa0009040 comp rx=0 tx=0).stop 2024-03-21T08:43:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 msgr2=0x7f5ab006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80ac040 tx=0x7f5aa801ace0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 shutdown_connections 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ab015bdc0 0x7f5ab006c280 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ab00cc640 0x7f5ab0158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5a8c067b20 0x7f5a8c069fe0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 --2- 172.21.15.28:0/2416167109 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ab009a3a0 0x7f5ab01fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 >> 172.21.15.28:0/2416167109 conn(0x7f5ab00b3620 msgr2=0x7f5ab0073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 shutdown_connections 2024-03-21T08:43:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:48.817+0000 7f5ab6400640 1 -- 172.21.15.28:0/2416167109 wait complete. 2024-03-21T08:43:48.840 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:48.840 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:48.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.025+0000 7ff358600640 1 Processor -- start 2024-03-21T08:43:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.025+0000 7ff358600640 1 -- start start 2024-03-21T08:43:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff358600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7ff35015c680 0x7ff3500ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff358600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff35006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff358600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3500ceaa0 con 0x7ff3500cfaa0 2024-03-21T08:43:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff358600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35006bb00 con 0x7ff35015c680 2024-03-21T08:43:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff358600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500ced80 con 0x7ff35006b170 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3500cfaa0 0x7ff35015ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36922/0 (socket says 172.21.15.28:36922) 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355200640 1 -- 172.21.15.28:0/3316319824 learned_addr learned my addr 172.21.15.28:0/3316319824 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355c00640 1 --2- 172.21.15.28:0/3316319824 >> v2:172.21.15.151:3300/0 conn(0x7ff35015c680 0x7ff3500ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff356600640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff35006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff34fe00640 1 -- 172.21.15.28:0/3316319824 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379890198 0 0) 0x7ff3500ceaa0 con 0x7ff3500cfaa0 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff34fe00640 1 -- 172.21.15.28:0/3316319824 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff338003530 con 0x7ff3500cfaa0 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355c00640 1 -- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 msgr2=0x7ff35006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355c00640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff35006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355c00640 1 -- 172.21.15.28:0/3316319824 >> v1:172.21.15.28:6789/0 conn(0x7ff3500cfaa0 legacy=0x7ff35015ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.029+0000 7ff355c00640 1 -- 172.21.15.28:0/3316319824 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500cf060 con 0x7ff35015c680 2024-03-21T08:43:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff355c00640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500ce4c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff3440097c0 tx=0x7ff34401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=365fa336282acc54 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff34fe00640 1 -- 172.21.15.28:0/3316319824 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440b0070 con 0x7ff35015c680 2024-03-21T08:43:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff34fe00640 1 -- 172.21.15.28:0/3316319824 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3440ab600 con 0x7ff35015c680 2024-03-21T08:43:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff34fe00640 1 -- 172.21.15.28:0/3316319824 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440ab9d0 con 0x7ff35015c680 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 -- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 msgr2=0x7ff3500ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500ce4c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff3440097c0 tx=0x7ff34401c410 comp rx=0 tx=0).stop 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 -- 172.21.15.28:0/3316319824 shutdown_connections 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff35006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 --2- 172.21.15.28:0/3316319824 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500ce4c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 -- 172.21.15.28:0/3316319824 >> 172.21.15.28:0/3316319824 conn(0x7ff350099a30 msgr2=0x7ff350099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 -- 172.21.15.28:0/3316319824 shutdown_connections 2024-03-21T08:43:49.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.033+0000 7ff358600640 1 -- 172.21.15.28:0/3316319824 wait complete. 2024-03-21T08:43:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 Processor -- start 2024-03-21T08:43:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- start start 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff3500706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500cd500 con 0x7ff3500cfaa0 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500cd380 con 0x7ff35015c680 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff350168aa0 con 0x7ff35006b170 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50736/0 (socket says 172.21.15.28:50736) 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 -- 172.21.15.28:0/54998627 learned_addr learned my addr 172.21.15.28:0/54998627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355c00640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff3500706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff356600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 -- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 msgr2=0x7ff3500706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff3500706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 -- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 msgr2=0x7ff3500b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 -- 172.21.15.28:0/54998627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500c5090 con 0x7ff3500cfaa0 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff355200640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0010f0 tx=0x7ff33c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c017070 con 0x7ff3500cfaa0 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c005d10 con 0x7ff3500cfaa0 2024-03-21T08:43:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3500c5370 con 0x7ff3500cfaa0 2024-03-21T08:43:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c007310 con 0x7ff3500cfaa0 2024-03-21T08:43:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.037+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff350152f20 con 0x7ff3500cfaa0 2024-03-21T08:43:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.041+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35004f410 con 0x7ff3500cfaa0 2024-03-21T08:43:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.041+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff33c01a070 con 0x7ff3500cfaa0 2024-03-21T08:43:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.041+0000 7ff34d600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 0x7ff330069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.041+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff33c085470 con 0x7ff3500cfaa0 2024-03-21T08:43:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.041+0000 7ff355c00640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 0x7ff330069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.045+0000 7ff355c00640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 0x7ff330069fe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3440ad000 tx=0x7ff3440a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.045+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff33c055600 con 0x7ff3500cfaa0 2024-03-21T08:43:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.213+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3500b6870 con 0x7ff3500cfaa0 2024-03-21T08:43:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff34d600640 1 -- 172.21.15.28:0/54998627 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7ff33c0596a0 con 0x7ff3500cfaa0 2024-03-21T08:43:49.219 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:49.219 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 msgr2=0x7ff330069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 0x7ff330069fe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff3440ad000 tx=0x7ff3440a6000 comp rx=0 tx=0).stop 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 msgr2=0x7ff3501529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0010f0 tx=0x7ff33c00c890 comp rx=0 tx=0).stop 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 shutdown_connections 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff35015c680 0x7ff3500b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3500cfaa0 0x7ff3501529e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff330067b20 0x7ff330069fe0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 --2- 172.21.15.28:0/54998627 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff35006b170 0x7ff3500706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 >> 172.21.15.28:0/54998627 conn(0x7ff350099a30 msgr2=0x7ff35015ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 shutdown_connections 2024-03-21T08:43:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.217+0000 7ff358600640 1 -- 172.21.15.28:0/54998627 wait complete. 2024-03-21T08:43:49.242 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:49.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:49.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 Processor -- start 2024-03-21T08:43:49.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 -- start start 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff5140cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff5140d0190 0x7ff51406c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51406d070 con 0x7ff51406cbe0 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5140d05f0 con 0x7ff5140d0190 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff519400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140b3010 con 0x7ff51415e1e0 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff5140d0190 0x7ff51406c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff5140d0190 0x7ff51406c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50750/0 (socket says 172.21.15.28:50750) 2024-03-21T08:43:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 -- 172.21.15.28:0/511843161 learned_addr learned my addr 172.21.15.28:0/511843161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff512800640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff5140cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 -- 172.21.15.28:0/511843161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 msgr2=0x7ff5140cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff5140cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 -- 172.21.15.28:0/511843161 >> v1:172.21.15.28:6789/0 conn(0x7ff51406cbe0 legacy=0x7ff514161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff509e00640 1 -- 172.21.15.28:0/511843161 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51406d310 con 0x7ff5140d0190 2024-03-21T08:43:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff512800640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff5140cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.385+0000 7ff511e00640 1 -- 172.21.15.28:0/511843161 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249604179 0 0) 0x7ff51406d070 con 0x7ff51406cbe0 2024-03-21T08:43:49.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff509e00640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff51406c6a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de83db01cb39743 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff511e00640 1 -- 172.21.15.28:0/511843161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0ab650 con 0x7ff5140d0190 2024-03-21T08:43:49.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff511e00640 1 -- 172.21.15.28:0/511843161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0b0070 con 0x7ff5140d0190 2024-03-21T08:43:49.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff511e00640 1 -- 172.21.15.28:0/511843161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0aba10 con 0x7ff5140d0190 2024-03-21T08:43:49.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- 172.21.15.28:0/511843161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 msgr2=0x7ff51406c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff51406c6a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc01c560 comp rx=0 tx=0).stop 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- 172.21.15.28:0/511843161 shutdown_connections 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff51406c6a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 --2- 172.21.15.28:0/511843161 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff5140cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- 172.21.15.28:0/511843161 >> 172.21.15.28:0/511843161 conn(0x7ff5140d1ab0 msgr2=0x7ff5140d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- 172.21.15.28:0/511843161 shutdown_connections 2024-03-21T08:43:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- 172.21.15.28:0/511843161 wait complete. 2024-03-21T08:43:49.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 Processor -- start 2024-03-21T08:43:49.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.389+0000 7ff519400640 1 -- start start 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 0x7ff514150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff51414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51416b0c0 con 0x7ff5140d0190 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51416af40 con 0x7ff51406cbe0 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51416b240 con 0x7ff51415e1e0 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff512800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 0x7ff514150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff513200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff51414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50764/0 (socket says 172.21.15.28:50764) 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff512800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 0x7ff514150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:35044/0 (socket says 172.21.15.28:35044) 2024-03-21T08:43:49.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 -- 172.21.15.28:0/3774975785 learned_addr learned my addr 172.21.15.28:0/3774975785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 -- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 msgr2=0x7ff51414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff51414a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 -- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 msgr2=0x7ff514150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 0x7ff514150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 -- 172.21.15.28:0/3774975785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140ca2c0 con 0x7ff5140d0190 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff509e00640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0ad040 tx=0x7ff4fc01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0b4600 con 0x7ff5140d0190 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0b47c0 con 0x7ff5140d0190 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5140ca5a0 con 0x7ff5140d0190 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0b4bb0 con 0x7ff5140d0190 2024-03-21T08:43:49.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5140cab60 con 0x7ff5140d0190 2024-03-21T08:43:49.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.393+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5140712c0 con 0x7ff5140d0190 2024-03-21T08:43:49.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.397+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff4fc0c7390 con 0x7ff5140d0190 2024-03-21T08:43:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.397+0000 7ff50b400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 0x7ff4f006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.397+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4fc1208c0 con 0x7ff5140d0190 2024-03-21T08:43:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.397+0000 7ff512800640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 0x7ff4f006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.397+0000 7ff512800640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 0x7ff4f006a010 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff50400ab60 tx=0x7ff504008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.401+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff4fc0c4060 con 0x7ff5140d0190 2024-03-21T08:43:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.713+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff5140d2830 con 0x7ff5140d0190 2024-03-21T08:43:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.713+0000 7ff50b400640 1 -- 172.21.15.28:0/3774975785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff4fc016150 con 0x7ff5140d0190 2024-03-21T08:43:49.718 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 msgr2=0x7ff4f006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 0x7ff4f006a010 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff50400ab60 tx=0x7ff504008040 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 msgr2=0x7ff514149b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0ad040 tx=0x7ff4fc01ada0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 shutdown_connections 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff4f0067b50 0x7ff4f006a010 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff51415e1e0 0x7ff51414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5140d0190 0x7ff514149b10 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 --2- 172.21.15.28:0/3774975785 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff51406cbe0 0x7ff514150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 >> 172.21.15.28:0/3774975785 conn(0x7ff5140d1ab0 msgr2=0x7ff5140708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 shutdown_connections 2024-03-21T08:43:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.717+0000 7ff519400640 1 -- 172.21.15.28:0/3774975785 wait complete. 2024-03-21T08:43:49.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:43:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 Processor -- start 2024-03-21T08:43:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 -- start start 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa80cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7effa80d0190 0x7effa806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa806d070 con 0x7effa806cbe0 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80d05f0 con 0x7effa80d0190 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effad200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80b3010 con 0x7effa815e1e0 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7eff9fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7effa80d0190 0x7effa806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7eff9fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7effa80d0190 0x7effa806c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50778/0 (socket says 172.21.15.28:50778) 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7eff9fe00640 1 -- 172.21.15.28:0/1051200892 learned_addr learned my addr 172.21.15.28:0/1051200892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.889+0000 7effa5e00640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa80cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7eff9fe00640 1 -- 172.21.15.28:0/1051200892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 msgr2=0x7effa80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7eff9fe00640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa80cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7eff9fe00640 1 -- 172.21.15.28:0/1051200892 >> v1:172.21.15.151:6789/0 conn(0x7effa806cbe0 legacy=0x7effa8161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7eff9fe00640 1 -- 172.21.15.28:0/1051200892 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa806d310 con 0x7effa80d0190 2024-03-21T08:43:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7effa5400640 1 -- 172.21.15.28:0/1051200892 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387397944 0 0) 0x7effa806d070 con 0x7effa806cbe0 2024-03-21T08:43:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7eff9fe00640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa806c6a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7eff90009870 tx=0x7eff9001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ab79af4cd75abb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7effa5400640 1 -- 172.21.15.28:0/1051200892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900ab650 con 0x7effa80d0190 2024-03-21T08:43:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7effa5400640 1 -- 172.21.15.28:0/1051200892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900b0070 con 0x7effa80d0190 2024-03-21T08:43:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.893+0000 7effa5400640 1 -- 172.21.15.28:0/1051200892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900aba10 con 0x7effa80d0190 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- 172.21.15.28:0/1051200892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 msgr2=0x7effa806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa806c6a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7eff90009870 tx=0x7eff9001c560 comp rx=0 tx=0).stop 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- 172.21.15.28:0/1051200892 shutdown_connections 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa806c6a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- 172.21.15.28:0/1051200892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa80cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- 172.21.15.28:0/1051200892 >> 172.21.15.28:0/1051200892 conn(0x7effa80d1ab0 msgr2=0x7effa80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- 172.21.15.28:0/1051200892 shutdown_connections 2024-03-21T08:43:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- 172.21.15.28:0/1051200892 wait complete. 2024-03-21T08:43:49.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 Processor -- start 2024-03-21T08:43:49.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- start start 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7effa806cbe0 0x7effa8161280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa8149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa816b0c0 con 0x7effa80d0190 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa816af40 con 0x7effa806cbe0 2024-03-21T08:43:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effad200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa816b240 con 0x7effa815e1e0 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50788/0 (socket says 172.21.15.28:50788) 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 -- 172.21.15.28:0/2438066333 learned_addr learned my addr 172.21.15.28:0/2438066333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effa5e00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7effa806cbe0 0x7effa8161280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7effa6800640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa8149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 -- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 msgr2=0x7effa8149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa8149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 -- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7effa806cbe0 msgr2=0x7effa8161280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7effa806cbe0 0x7effa8161280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.897+0000 7eff9fe00640 1 -- 172.21.15.28:0/2438066333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa815ec40 con 0x7effa80d0190 2024-03-21T08:43:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9fe00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7eff9001a190 tx=0x7eff9001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900ba070 con 0x7effa80d0190 2024-03-21T08:43:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90004230 con 0x7effa80d0190 2024-03-21T08:43:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa815eec0 con 0x7effa80d0190 2024-03-21T08:43:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900ab8a0 con 0x7effa80d0190 2024-03-21T08:43:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa815f480 con 0x7effa80d0190 2024-03-21T08:43:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80d1860 con 0x7effa80d0190 2024-03-21T08:43:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7eff900aba40 con 0x7effa80d0190 2024-03-21T08:43:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 0x7eff84069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7eff90120210 con 0x7effa80d0190 2024-03-21T08:43:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.901+0000 7effa5e00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 0x7eff84069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.905+0000 7effa5e00640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 0x7eff84069fe0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7effa808b7a0 tx=0x7eff98009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:49.905+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7effa815ec40 con 0x7effa80d0190 2024-03-21T08:43:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.073+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7effa80d2830 con 0x7effa80d0190 2024-03-21T08:43:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.073+0000 7eff9e000640 1 -- 172.21.15.28:0/2438066333 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7eff90016150 con 0x7effa80d0190 2024-03-21T08:43:50.078 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 msgr2=0x7eff84069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 0x7eff84069fe0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7effa808b7a0 tx=0x7eff98009190 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 msgr2=0x7effa815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7eff9001a190 tx=0x7eff9001ada0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 shutdown_connections 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7eff84067b20 0x7eff84069fe0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7effa815e1e0 0x7effa8149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effa80d0190 0x7effa815e700 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 --2- 172.21.15.28:0/2438066333 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7effa806cbe0 0x7effa8161280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 >> 172.21.15.28:0/2438066333 conn(0x7effa80d1ab0 msgr2=0x7effa808ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 shutdown_connections 2024-03-21T08:43:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.077+0000 7effad200640 1 -- 172.21.15.28:0/2438066333 wait complete. 2024-03-21T08:43:50.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:43:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.261+0000 7f3877e00640 1 Processor -- start 2024-03-21T08:43:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.261+0000 7f3877e00640 1 -- start start 2024-03-21T08:43:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f387015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f3870099a30 0x7f387015eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f387015f040 0x7f387015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38700d0c60 con 0x7f387015f040 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38700d0f40 con 0x7f3870099a30 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3877e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387006c320 con 0x7f387015a990 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f387015f040 0x7f387015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f387015f040 0x7f387015ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50798/0 (socket says 172.21.15.28:50798) 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 -- 172.21.15.28:0/2367625805 learned_addr learned my addr 172.21.15.28:0/2367625805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3874a00640 1 --2- 172.21.15.28:0/2367625805 >> v2:172.21.15.151:3300/0 conn(0x7f3870099a30 0x7f387015eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875400640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f387015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 -- 172.21.15.28:0/2367625805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 msgr2=0x7f387015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f387015e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 -- 172.21.15.28:0/2367625805 >> v2:172.21.15.151:3300/0 conn(0x7f3870099a30 msgr2=0x7f387015eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 --2- 172.21.15.28:0/2367625805 >> v2:172.21.15.151:3300/0 conn(0x7f3870099a30 0x7f387015eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.265+0000 7f3875e00640 1 -- 172.21.15.28:0/2367625805 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387015f470 con 0x7f387015f040 2024-03-21T08:43:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3875e00640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f387015f040 0x7f387015ad70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f387006dc20 tx=0x7f386801ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773a30e9bddba42f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f386fe00640 1 -- 172.21.15.28:0/2367625805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386800b2d0 con 0x7f387015f040 2024-03-21T08:43:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f386fe00640 1 -- 172.21.15.28:0/2367625805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386800b470 con 0x7f387015f040 2024-03-21T08:43:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f386fe00640 1 -- 172.21.15.28:0/2367625805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38680a7980 con 0x7f387015f040 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- 172.21.15.28:0/2367625805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f387015f040 msgr2=0x7f387015ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f387015f040 0x7f387015ad70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f387006dc20 tx=0x7f386801ead0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- 172.21.15.28:0/2367625805 shutdown_connections 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f387015f040 0x7f387015ad70 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 --2- 172.21.15.28:0/2367625805 >> v2:172.21.15.151:3300/0 conn(0x7f3870099a30 0x7f387015eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 --2- 172.21.15.28:0/2367625805 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f387015e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- 172.21.15.28:0/2367625805 >> 172.21.15.28:0/2367625805 conn(0x7f3870072860 msgr2=0x7f38700b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- 172.21.15.28:0/2367625805 shutdown_connections 2024-03-21T08:43:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- 172.21.15.28:0/2367625805 wait complete. 2024-03-21T08:43:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 Processor -- start 2024-03-21T08:43:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.269+0000 7f3877e00640 1 -- start start 2024-03-21T08:43:50.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f38701556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f387015f040 0x7f387009bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387015be50 con 0x7f3870099a30 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387015bcd0 con 0x7f387015f040 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387006b170 con 0x7f387015a990 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50808/0 (socket says 172.21.15.28:50808) 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 -- 172.21.15.28:0/3678831202 learned_addr learned my addr 172.21.15.28:0/3678831202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3874a00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f38701556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f387015f040 0x7f387009bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 -- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 msgr2=0x7f38701556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f38701556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 -- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f387015f040 msgr2=0x7f387009bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f387015f040 0x7f387009bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 -- 172.21.15.28:0/3678831202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3870159860 con 0x7f3870099a30 2024-03-21T08:43:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3875400640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f386400e960 tx=0x7f386400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864017070 con 0x7f3870099a30 2024-03-21T08:43:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386400ca00 con 0x7f3870099a30 2024-03-21T08:43:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386400cdb0 con 0x7f3870099a30 2024-03-21T08:43:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3870158ce0 con 0x7f3870099a30 2024-03-21T08:43:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3870159580 con 0x7f3870099a30 2024-03-21T08:43:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387004f4f0 con 0x7f3870099a30 2024-03-21T08:43:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.273+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f386401a070 con 0x7f3870099a30 2024-03-21T08:43:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.277+0000 7f386d600640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 0x7f3850069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.277+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3864085ff0 con 0x7f3870099a30 2024-03-21T08:43:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.277+0000 7f3874a00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 0x7f3850069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.277+0000 7f3874a00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 0x7f3850069fe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f38700cf480 tx=0x7f385c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.281+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3864056250 con 0x7f3870099a30 2024-03-21T08:43:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.441+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f38700af020 con 0x7f3870099a30 2024-03-21T08:43:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f386d600640 1 -- 172.21.15.28:0/3678831202 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f386405a2f0 con 0x7f3870099a30 2024-03-21T08:43:50.481 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 msgr2=0x7f3850069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 0x7f3850069fe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f38700cf480 tx=0x7f385c009190 comp rx=0 tx=0).stop 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 msgr2=0x7f3870154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f386400e960 tx=0x7f386400ee20 comp rx=0 tx=0).stop 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 shutdown_connections 2024-03-21T08:43:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f387015f040 0x7f387009bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3850067b20 0x7f3850069fe0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f387015a990 0x7f38701556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 --2- 172.21.15.28:0/3678831202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3870099a30 0x7f3870154ec0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 >> 172.21.15.28:0/3678831202 conn(0x7f3870072860 msgr2=0x7f38700af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 shutdown_connections 2024-03-21T08:43:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.445+0000 7f3877e00640 1 -- 172.21.15.28:0/3678831202 wait complete. 2024-03-21T08:43:50.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:43:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.621+0000 7f4c5c400640 1 Processor -- start 2024-03-21T08:43:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.621+0000 7f4c5c400640 1 -- start start 2024-03-21T08:43:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5c400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 0x7f4c540ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5c400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5415fba0 con 0x7f4c5415bd20 2024-03-21T08:43:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5c400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540cee30 con 0x7f4c540b0e80 2024-03-21T08:43:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415fe80 con 0x7f4c540cea00 2024-03-21T08:43:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c59000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 0x7f4c540ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:52558/0 (socket says 172.21.15.28:52558) 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c59a00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f4c5415bd20 0x7f4c540b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:41626/0 (socket says 172.21.15.28:41626) 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 -- 172.21.15.28:0/3320201769 learned_addr learned my addr 172.21.15.28:0/3320201769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 -- 172.21.15.28:0/3320201769 >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 msgr2=0x7f4c540ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 --2- 172.21.15.28:0/3320201769 >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 0x7f4c540ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 -- 172.21.15.28:0/3320201769 >> v1:172.21.15.151:6789/0 conn(0x7f4c5415bd20 legacy=0x7f4c540b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c5a400640 1 -- 172.21.15.28:0/3320201769 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54160160 con 0x7f4c540cea00 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c59000640 1 --2- 172.21.15.28:0/3320201769 >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 0x7f4c540ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.625+0000 7f4c53e00640 1 -- 172.21.15.28:0/3320201769 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225867253 0 0) 0x7f4c5415fba0 con 0x7f4c5415bd20 2024-03-21T08:43:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5a400640 1 --2- 172.21.15.28:0/3320201769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009e50 tx=0x7f4c4c01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a653ca56f74964 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c53e00640 1 -- 172.21.15.28:0/3320201769 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0a5700 con 0x7f4c540cea00 2024-03-21T08:43:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c53e00640 1 -- 172.21.15.28:0/3320201769 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c0a58a0 con 0x7f4c540cea00 2024-03-21T08:43:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c53e00640 1 -- 172.21.15.28:0/3320201769 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0a5c70 con 0x7f4c540cea00 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- 172.21.15.28:0/3320201769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 msgr2=0x7f4c5408afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 --2- 172.21.15.28:0/3320201769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009e50 tx=0x7f4c4c01ca30 comp rx=0 tx=0).stop 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- 172.21.15.28:0/3320201769 shutdown_connections 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 --2- 172.21.15.28:0/3320201769 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540cea00 0x7f4c5408afc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 --2- 172.21.15.28:0/3320201769 >> v2:172.21.15.151:3300/0 conn(0x7f4c540b0e80 0x7f4c540ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- 172.21.15.28:0/3320201769 >> 172.21.15.28:0/3320201769 conn(0x7f4c540d1ab0 msgr2=0x7f4c540d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- 172.21.15.28:0/3320201769 shutdown_connections 2024-03-21T08:43:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- 172.21.15.28:0/3320201769 wait complete. 2024-03-21T08:43:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 Processor -- start 2024-03-21T08:43:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.629+0000 7f4c5c400640 1 -- start start 2024-03-21T08:43:50.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540b0e80 0x7f4c540c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 0x7f4c5414ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5408bd10 con 0x7f4c5415bd20 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415c310 con 0x7f4c540cea00 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5408be90 con 0x7f4c540b0e80 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53196/0 (socket says 172.21.15.28:53196) 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 -- 172.21.15.28:0/324159048 learned_addr learned my addr 172.21.15.28:0/324159048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c59a00640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540b0e80 0x7f4c540c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c59000640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 0x7f4c5414ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 -- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540b0e80 msgr2=0x7f4c540c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540b0e80 0x7f4c540c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 -- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 msgr2=0x7f4c5414ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 0x7f4c5414ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 -- 172.21.15.28:0/324159048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540c7480 con 0x7f4c5415bd20 2024-03-21T08:43:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c59000640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 0x7f4c5414ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5a400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0af040 tx=0x7f4c4c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0bc020 con 0x7f4c5415bd20 2024-03-21T08:43:50.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c0b8670 con 0x7f4c5415bd20 2024-03-21T08:43:50.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c009510 con 0x7f4c5415bd20 2024-03-21T08:43:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c540c7760 con 0x7f4c5415bd20 2024-03-21T08:43:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c540c7d20 con 0x7f4c5415bd20 2024-03-21T08:43:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c540b0c60 con 0x7f4c5415bd20 2024-03-21T08:43:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.633+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f4c4c0bf020 con 0x7f4c5415bd20 2024-03-21T08:43:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.637+0000 7f4c51600640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 0x7f4c34069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.637+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c4c120ec0 con 0x7f4c5415bd20 2024-03-21T08:43:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.637+0000 7f4c59a00640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 0x7f4c34069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:50.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.637+0000 7f4c59a00640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 0x7f4c34069fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c48007be0 tx=0x7f4c48007b30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.645+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4c4c0c6030 con 0x7f4c5415bd20 2024-03-21T08:43:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.817+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4c5404f4f0 con 0x7f4c5415bd20 2024-03-21T08:43:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.817+0000 7f4c51600640 1 -- 172.21.15.28:0/324159048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4c4c016620 con 0x7f4c5415bd20 2024-03-21T08:43:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 msgr2=0x7f4c34069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 0x7f4c34069fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c48007be0 tx=0x7f4c48007b30 comp rx=0 tx=0).stop 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 msgr2=0x7f4c5414ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0af040 tx=0x7f4c4c0197d0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 shutdown_connections 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5415bd20 0x7f4c5414ef50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4c540cea00 0x7f4c5414ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f4c34067b20 0x7f4c34069fe0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 --2- 172.21.15.28:0/324159048 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f4c540b0e80 0x7f4c540c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 >> 172.21.15.28:0/324159048 conn(0x7f4c540d1ab0 msgr2=0x7f4c54073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 shutdown_connections 2024-03-21T08:43:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:50.821+0000 7f4c5c400640 1 -- 172.21.15.28:0/324159048 wait complete. 2024-03-21T08:43:50.843 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:43:50.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:50.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:50.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40323]: fuse finished with error 0 and tester_r 0 2024-03-21T08:43:50.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:43:56.966 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:43:56.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:56.967 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:43:56.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 Processor -- start 2024-03-21T08:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 -- start start 2024-03-21T08:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 0x7facec06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec161ec0 con 0x7facec15dd60 2024-03-21T08:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec162750 con 0x7facec0ce400 2024-03-21T08:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7faceb200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0cdf70 con 0x7facec0ce880 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7face1400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 0x7facec06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7facea800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7face9e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7facec15dd60 0x7facec1617b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:41630/0 (socket says 172.21.15.28:41630) 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7face9e00640 1 -- 172.21.15.28:0/400278875 learned_addr learned my addr 172.21.15.28:0/400278875 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7facea800640 1 -- 172.21.15.28:0/400278875 >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 msgr2=0x7facec06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7facea800640 1 --2- 172.21.15.28:0/400278875 >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 0x7facec06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7facea800640 1 -- 172.21.15.28:0/400278875 >> v1:172.21.15.151:6789/0 conn(0x7facec15dd60 legacy=0x7facec1617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7facea800640 1 -- 172.21.15.28:0/400278875 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec162190 con 0x7facec0ce880 2024-03-21T08:43:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7face1400640 1 --2- 172.21.15.28:0/400278875 >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 0x7facec06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:43:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.161+0000 7face9400640 1 -- 172.21.15.28:0/400278875 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409560499 0 0) 0x7facec161ec0 con 0x7facec15dd60 2024-03-21T08:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7facea800640 1 --2- 172.21.15.28:0/400278875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06b6b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7face40097c0 tx=0x7face401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13244bd11b7ec4b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7face9400640 1 -- 172.21.15.28:0/400278875 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face40b0070 con 0x7facec0ce880 2024-03-21T08:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7face9400640 1 -- 172.21.15.28:0/400278875 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face40ab600 con 0x7facec0ce880 2024-03-21T08:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7face9400640 1 -- 172.21.15.28:0/400278875 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face40ab9d0 con 0x7facec0ce880 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 -- 172.21.15.28:0/400278875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 msgr2=0x7facec06b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 --2- 172.21.15.28:0/400278875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06b6b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7face40097c0 tx=0x7face401c380 comp rx=0 tx=0).stop 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 -- 172.21.15.28:0/400278875 shutdown_connections 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 --2- 172.21.15.28:0/400278875 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06b6b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 --2- 172.21.15.28:0/400278875 >> v2:172.21.15.151:3300/0 conn(0x7facec0ce400 0x7facec06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 -- 172.21.15.28:0/400278875 >> 172.21.15.28:0/400278875 conn(0x7facec0cc980 msgr2=0x7facec0ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 -- 172.21.15.28:0/400278875 shutdown_connections 2024-03-21T08:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.165+0000 7faceb200640 1 -- 172.21.15.28:0/400278875 wait complete. 2024-03-21T08:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 Processor -- start 2024-03-21T08:43:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- start start 2024-03-21T08:43:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7facec15dd60 0x7facec0a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec16acd0 con 0x7facec0ce400 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec16ab50 con 0x7facec15dd60 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec16ae50 con 0x7facec0ce880 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53204/0 (socket says 172.21.15.28:53204) 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 -- 172.21.15.28:0/937789892 learned_addr learned my addr 172.21.15.28:0/937789892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face1400640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7facea800640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7facec15dd60 0x7facec0a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 -- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 msgr2=0x7facec06f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 -- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7facec15dd60 msgr2=0x7facec0a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7facec15dd60 0x7facec0a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 -- 172.21.15.28:0/937789892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec06e6a0 con 0x7facec0ce400 2024-03-21T08:43:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face9e00640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c7f0 tx=0x7facdc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc018070 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00a640 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec06e980 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00aa30 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.169+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec06ef40 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.173+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec075270 con 0x7facec0ce400 2024-03-21T08:43:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.173+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7facdc01b070 con 0x7facec0ce400 2024-03-21T08:43:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.173+0000 7face2a00640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 0x7facc8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.173+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7facdc085680 con 0x7facec0ce400 2024-03-21T08:43:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.173+0000 7face1400640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 0x7facc8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.177+0000 7face1400640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 0x7facc8069fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7facd4009580 tx=0x7facd4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.177+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7facdc0558a0 con 0x7facec0ce400 2024-03-21T08:43:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.381+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7facec0c3780 con 0x7facec0ce400 2024-03-21T08:43:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.381+0000 7face2a00640 1 -- 172.21.15.28:0/937789892 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7facdc059940 con 0x7facec0ce400 2024-03-21T08:43:57.385 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 msgr2=0x7facc8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 0x7facc8069fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7facd4009580 tx=0x7facd4009190 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 msgr2=0x7facec1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c7f0 tx=0x7facdc00ccb0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 shutdown_connections 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7facec15dd60 0x7facec0a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7facc8067b20 0x7facc8069fe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7facec0ce880 0x7facec06f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 --2- 172.21.15.28:0/937789892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec0ce400 0x7facec1598d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 >> 172.21.15.28:0/937789892 conn(0x7facec0cc980 msgr2=0x7facec06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 shutdown_connections 2024-03-21T08:43:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:57.385+0000 7faceb200640 1 -- 172.21.15.28:0/937789892 wait complete. 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:43:57.405 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:57.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.406 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:43:57.414 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42232sec preferred_lft 42232sec 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:43:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:43:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:43:57.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.418 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:43:57.480 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:43:57.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.483 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:43:57.535 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:43:57.535 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:43:57.535 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:43:57.535 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:43:57.536 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:43:57.536 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:43:57.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:43:57.536 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:43:57.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.627 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:43:57.635 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:43:57.636 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:43:57.636 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:57.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:43:57.702 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:43:57.702 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:57.702 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:57.754 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:43:57.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:43:57.755 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:57.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:57.867 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:57.919 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:57.919 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:57.919 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:57.919 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:57.920 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:57.921 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:57.922 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:57.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:57.923 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:57.974 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:43:57.974 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:43:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:43:58.037 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:43:58.089 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:43:58.089 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:43:58.089 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:43:58.089 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:43:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:43:58.091 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:58.092 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:43:58.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:58.093 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:43:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:43:58.125+0000 7f568c8ce580 -1 init, newargv = 0x555f13c795e0 newargc=15 2024-03-21T08:43:58.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40723]: starting ceph client 2024-03-21T08:43:58.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40723]: starting fuse 2024-03-21T08:43:58.139 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:43:58.140 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:58.140 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:43:58.141 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:43:58.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.40723.asok 2024-03-21T08:43:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:43:58.277 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40723.asok 2024-03-21T08:43:58.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:58.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40723.asok status 2024-03-21T08:43:58.450 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:43:58.451 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "40723", 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6504, 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6504 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2069567036 2024-03-21T08:43:58.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2069567036 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.6504 192.168.0.1:0/2069567036", 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/2069567036", 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:43:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 28, 2024-03-21T08:43:58.454 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 54, 2024-03-21T08:43:58.454 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:43:58.454 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:43:58.454 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:43:58.454 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:43:58.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:58.465 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:43:58.465 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:58.474 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:43:58.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:58.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:43:58.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:43:58.537 DEBUG:teuthology.orchestra.run.smithi028:> stdin-killer -- python3 -c ' 2024-03-21T08:43:58.537 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:43:58.537 DEBUG:teuthology.orchestra.run.smithi028:> import errno 2024-03-21T08:43:58.537 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> ret = os.read(fd, 1024) 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> else: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> os.write(fd, b'"'"'content'"'"') 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> if e.errno != errno.EPERM: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> raise 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> else: 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("client does not check permission of data pool") 2024-03-21T08:43:58.538 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:43:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:43:58.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-21T08:43:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.849+0000 7f912fe00640 1 Processor -- start 2024-03-21T08:43:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.849+0000 7f912fe00640 1 -- start start 2024-03-21T08:43:58.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912fe00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f912815bd20 0x7f91280b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f912808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912815fba0 con 0x7f91280b0e80 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912fe00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912815fe80 con 0x7f912815bd20 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9128160160 con 0x7f91280cebd0 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91280b0e80 0x7f91280ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35168/0 (socket says 172.21.15.28:35168) 2024-03-21T08:43:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 -- 172.21.15.28:0/2528954711 learned_addr learned my addr 172.21.15.28:0/2528954711 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912d400640 1 --2- 172.21.15.28:0/2528954711 >> v2:172.21.15.151:3300/0 conn(0x7f912815bd20 0x7f91280b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912de00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f912808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206468219 0 0) 0x7f912815fba0 con 0x7f91280b0e80 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9110003530 con 0x7f91280b0e80 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788340375 0 0) 0x7f9110003530 con 0x7f91280b0e80 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 msgr2=0x7f912808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f912808afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 >> v2:172.21.15.151:3300/0 conn(0x7f912815bd20 msgr2=0x7f91280b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 --2- 172.21.15.28:0/2528954711 >> v2:172.21.15.151:3300/0 conn(0x7f912815bd20 0x7f91280b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128160440 con 0x7f91280b0e80 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f9114001680 con 0x7f91280b0e80 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 >> v1:172.21.15.28:6789/0 conn(0x7f91280b0e80 legacy=0x7f91280ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f91100039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9110003f40 0x7f9110004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9110004d50 0x7f91100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912815fe80 con 0x7f9110003f40 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128160160 con 0x7f9110004d50 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911000b9d0 con 0x7f91280cebd0 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9110003f40 0x7f9110004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912d400640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f91100039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912de00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9110004d50 0x7f91100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 -- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 msgr2=0x7f91100039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f91280cebd0 0x7f91100039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 -- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9110004d50 msgr2=0x7f91100051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9110004d50 0x7f91100051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.853+0000 7f912ca00640 1 -- 172.21.15.28:0/2528954711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128160440 con 0x7f9110003f40 2024-03-21T08:43:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912ca00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9110003f40 0x7f9110004810 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f911400c260 tx=0x7f911401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13573c7f3b6474f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114007d80 con 0x7f9110003f40 2024-03-21T08:43:58.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9114005b00 con 0x7f9110003f40 2024-03-21T08:43:58.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f9127e00640 1 -- 172.21.15.28:0/2528954711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114007070 con 0x7f9110003f40 2024-03-21T08:43:58.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- 172.21.15.28:0/2528954711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9110003f40 msgr2=0x7f9110004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:58.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 --2- 172.21.15.28:0/2528954711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9110003f40 0x7f9110004810 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f911400c260 tx=0x7f911401f3d0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912de00640 1 -- 172.21.15.28:0/2528954711 reap_dead start 2024-03-21T08:43:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- 172.21.15.28:0/2528954711 shutdown_connections 2024-03-21T08:43:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- 172.21.15.28:0/2528954711 >> 172.21.15.28:0/2528954711 conn(0x7f91280d1ab0 msgr2=0x7f91280d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- 172.21.15.28:0/2528954711 shutdown_connections 2024-03-21T08:43:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- 172.21.15.28:0/2528954711 wait complete. 2024-03-21T08:43:58.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 Processor -- start 2024-03-21T08:43:58.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.857+0000 7f912fe00640 1 -- start start 2024-03-21T08:43:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 0x7f91281f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 0x7f912808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280cc6a0 con 0x7f9128148eb0 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280cc520 con 0x7f912815a270 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912815c100 con 0x7f9110003f40 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53214/0 (socket says 172.21.15.28:53214) 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 -- 172.21.15.28:0/1477834274 learned_addr learned my addr 172.21.15.28:0/1477834274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912d400640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 0x7f91281f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912de00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 0x7f912808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 -- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 msgr2=0x7f91281f8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 0x7f91281f8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 -- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 msgr2=0x7f912808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 0x7f912808afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 -- 172.21.15.28:0/1477834274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128159760 con 0x7f9128148eb0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912d400640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 0x7f91281f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912de00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 0x7f912808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912ca00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f911401fc70 tx=0x7f9110005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114005d70 con 0x7f9128148eb0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91140047f0 con 0x7f9128148eb0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114004bc0 con 0x7f9128148eb0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9128159a40 con 0x7f9128148eb0 2024-03-21T08:43:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91280b08a0 con 0x7f9128148eb0 2024-03-21T08:43:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.861+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91280b0c80 con 0x7f9128148eb0 2024-03-21T08:43:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.865+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f91140ba030 con 0x7f9128148eb0 2024-03-21T08:43:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.865+0000 7f9125600640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 0x7f9108069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.865+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9114122c80 con 0x7f9128148eb0 2024-03-21T08:43:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.865+0000 7f912d400640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 0x7f9108069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.865+0000 7f912d400640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 0x7f9108069fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f911c009ad0 tx=0x7f911c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:58.869+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f91140c1030 con 0x7f9128148eb0 2024-03-21T08:43:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.089+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) v1 -- 0x7f91280ce120 con 0x7f9128148eb0 2024-03-21T08:43:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.089+0000 7f9125600640 1 -- 172.21.15.28:0/1477834274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 183+0+140 (secure 0 0 0) 0x7f9114019140 con 0x7f9128148eb0 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-21T08:43:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 msgr2=0x7f9108069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 0x7f9108069fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f911c009ad0 tx=0x7f911c008040 comp rx=0 tx=0).stop 2024-03-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 msgr2=0x7f9128159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f911401fc70 tx=0x7f9110005840 comp rx=0 tx=0).stop 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 shutdown_connections 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f912815a270 0x7f912808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9128148eb0 0x7f9128159d30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9108067b20 0x7f9108069fe0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 --2- 172.21.15.28:0/1477834274 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9110003f40 0x7f91281f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 >> 172.21.15.28:0/1477834274 conn(0x7f91280d1ab0 msgr2=0x7f9128073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 shutdown_connections 2024-03-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.093+0000 7f912fe00640 1 -- 172.21.15.28:0/1477834274 wait complete. 2024-03-21T08:43:59.123 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:43:59.123 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:43:59.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:43:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.309+0000 7f5ed9200640 1 Processor -- start 2024-03-21T08:43:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 -- start start 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed415bbe0 0x7f5ed415f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed40cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5ed40cf2f0 0x7f5ed40cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed4160590 con 0x7f5ed415bbe0 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed415ffd0 con 0x7f5ed40cf2f0 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed9200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed41602b0 con 0x7f5ed415fb50 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed415bbe0 0x7f5ed415f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed415bbe0 0x7f5ed415f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53220/0 (socket says 172.21.15.28:53220) 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 -- 172.21.15.28:0/2709654931 learned_addr learned my addr 172.21.15.28:0/2709654931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ecbe00640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed40cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed2800640 1 --2- 172.21.15.28:0/2709654931 >> v2:172.21.15.151:3300/0 conn(0x7f5ed40cf2f0 0x7f5ed40cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 -- 172.21.15.28:0/2709654931 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 msgr2=0x7f5ed40cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed40cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 -- 172.21.15.28:0/2709654931 >> v2:172.21.15.151:3300/0 conn(0x7f5ed40cf2f0 msgr2=0x7f5ed40cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/2709654931 >> v2:172.21.15.151:3300/0 conn(0x7f5ed40cf2f0 0x7f5ed40cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.313+0000 7f5ed1e00640 1 -- 172.21.15.28:0/2709654931 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40ce5b0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed415f610 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5ed406c230 tx=0x7f5ec401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a61ec64dc63ad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed1400640 1 -- 172.21.15.28:0/2709654931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40b1070 con 0x7f5ed415bbe0 2024-03-21T08:43:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed1400640 1 -- 172.21.15.28:0/2709654931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec40ac620 con 0x7f5ed415bbe0 2024-03-21T08:43:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed1400640 1 -- 172.21.15.28:0/2709654931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40ac9f0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 -- 172.21.15.28:0/2709654931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 msgr2=0x7f5ed415f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed415f610 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5ed406c230 tx=0x7f5ec401c4f0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 -- 172.21.15.28:0/2709654931 shutdown_connections 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 --2- 172.21.15.28:0/2709654931 >> v2:172.21.15.151:3300/0 conn(0x7f5ed40cf2f0 0x7f5ed40cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed40cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 --2- 172.21.15.28:0/2709654931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed415f610 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 -- 172.21.15.28:0/2709654931 >> 172.21.15.28:0/2709654931 conn(0x7f5ed4099a30 msgr2=0x7f5ed4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 -- 172.21.15.28:0/2709654931 shutdown_connections 2024-03-21T08:43:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.317+0000 7f5ed9200640 1 -- 172.21.15.28:0/2709654931 wait complete. 2024-03-21T08:43:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 Processor -- start 2024-03-21T08:43:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- start start 2024-03-21T08:43:59.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ed40cf2f0 0x7f5ed4165e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed4145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415c350 con 0x7f5ed415bbe0 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415c1d0 con 0x7f5ed40cf2f0 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed415c4d0 con 0x7f5ed415fb50 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53224/0 (socket says 172.21.15.28:53224) 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 -- 172.21.15.28:0/1426798636 learned_addr learned my addr 172.21.15.28:0/1426798636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed2800640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed4145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ed40cf2f0 0x7f5ed4165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 -- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 msgr2=0x7f5ed4145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed4145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 -- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ed40cf2f0 msgr2=0x7f5ed4165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ed40cf2f0 0x7f5ed4165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 -- 172.21.15.28:0/1426798636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4164dc0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ecbe00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00d5f0 tx=0x7f5ebc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0125f0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc019070 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0129b0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed41650a0 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.321+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed4165660 con 0x7f5ed415bbe0 2024-03-21T08:43:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.325+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed404f410 con 0x7f5ed415bbe0 2024-03-21T08:43:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.325+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5ebc012b50 con 0x7f5ed415bbe0 2024-03-21T08:43:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.325+0000 7f5eca000640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 0x7f5eb006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.325+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ebc086330 con 0x7f5ed415bbe0 2024-03-21T08:43:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.325+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 0x7f5eb006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.329+0000 7f5ed1e00640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 0x7f5eb006a010 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5ec40ae000 tx=0x7f5ec40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.329+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5ebc056560 con 0x7f5ed415bbe0 2024-03-21T08:43:59.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.501+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed40c7730 con 0x7f5ed415bbe0 2024-03-21T08:43:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5eca000640 1 -- 172.21.15.28:0/1426798636 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f5ebc05a600 con 0x7f5ed415bbe0 2024-03-21T08:43:59.507 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:43:59.508 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:43:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 msgr2=0x7f5eb006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 0x7f5eb006a010 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5ec40ae000 tx=0x7f5ec40a7000 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 msgr2=0x7f5ed4145540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00d5f0 tx=0x7f5ebc00dab0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 shutdown_connections 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5eb0067b50 0x7f5eb006a010 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ed415fb50 0x7f5ed4145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed415bbe0 0x7f5ed4145540 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 --2- 172.21.15.28:0/1426798636 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ed40cf2f0 0x7f5ed4165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 >> 172.21.15.28:0/1426798636 conn(0x7f5ed4099a30 msgr2=0x7f5ed415eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 shutdown_connections 2024-03-21T08:43:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.505+0000 7f5ed9200640 1 -- 172.21.15.28:0/1426798636 wait complete. 2024-03-21T08:43:59.529 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:43:59.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:43:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 Processor -- start 2024-03-21T08:43:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 -- start start 2024-03-21T08:43:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66180cc170 0x7f6618071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f66180ce320 0x7f66180ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66180ccea0 con 0x7f66180cc170 2024-03-21T08:43:59.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661815be30 con 0x7f66180ce320 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180cc8e0 con 0x7f661815ba50 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66180cc170 0x7f6618071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66180cc170 0x7f6618071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53226/0 (socket says 172.21.15.28:53226) 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 -- 172.21.15.28:0/2496626541 learned_addr learned my addr 172.21.15.28:0/2496626541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6617400640 1 --2- 172.21.15.28:0/2496626541 >> v2:172.21.15.151:3300/0 conn(0x7f66180ce320 0x7f66180ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f6616a00640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 -- 172.21.15.28:0/2496626541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 msgr2=0x7f66180b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 -- 172.21.15.28:0/2496626541 >> v2:172.21.15.151:3300/0 conn(0x7f66180ce320 msgr2=0x7f66180ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:43:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 --2- 172.21.15.28:0/2496626541 >> v2:172.21.15.151:3300/0 conn(0x7f66180ce320 0x7f66180ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.677+0000 7f660e000640 1 -- 172.21.15.28:0/2496626541 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66180ccbc0 con 0x7f66180cc170 2024-03-21T08:43:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f660e000640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618071bb0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6600009870 tx=0x7f660001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fa1dbf0187ac35 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6616000640 1 -- 172.21.15.28:0/2496626541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000b1070 con 0x7f66180cc170 2024-03-21T08:43:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6616000640 1 -- 172.21.15.28:0/2496626541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000ac630 con 0x7f66180cc170 2024-03-21T08:43:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6616000640 1 -- 172.21.15.28:0/2496626541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000a8050 con 0x7f66180cc170 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 -- 172.21.15.28:0/2496626541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 msgr2=0x7f6618071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618071bb0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6600009870 tx=0x7f660001c520 comp rx=0 tx=0).stop 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 -- 172.21.15.28:0/2496626541 shutdown_connections 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 --2- 172.21.15.28:0/2496626541 >> v2:172.21.15.151:3300/0 conn(0x7f66180ce320 0x7f66180ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618071bb0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 --2- 172.21.15.28:0/2496626541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 -- 172.21.15.28:0/2496626541 >> 172.21.15.28:0/2496626541 conn(0x7f661806fa90 msgr2=0x7f661806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 -- 172.21.15.28:0/2496626541 shutdown_connections 2024-03-21T08:43:59.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.681+0000 7f6617e00640 1 -- 172.21.15.28:0/2496626541 wait complete. 2024-03-21T08:43:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 Processor -- start 2024-03-21T08:43:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- start start 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 0x7f6618159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66180b5120 con 0x7f66180cc170 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66180720f0 con 0x7f66180ce320 2024-03-21T08:43:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180b52a0 con 0x7f661815ba50 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53242/0 (socket says 172.21.15.28:53242) 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 -- 172.21.15.28:0/1233072299 learned_addr learned my addr 172.21.15.28:0/1233072299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617400640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f660e000640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 0x7f6618159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 -- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 msgr2=0x7f66180c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 -- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 msgr2=0x7f6618159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 0x7f6618159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 -- 172.21.15.28:0/1233072299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618151a40 con 0x7f66180cc170 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f660e000640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 0x7f6618159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:43:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6616a00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f660800e930 tx=0x7f660800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608017070 con 0x7f66180cc170 2024-03-21T08:43:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660800ca00 con 0x7f66180cc170 2024-03-21T08:43:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66180c6740 con 0x7f66180cc170 2024-03-21T08:43:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800cdb0 con 0x7f66180cc170 2024-03-21T08:43:59.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.685+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661815dc90 con 0x7f66180cc170 2024-03-21T08:43:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.689+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661804f410 con 0x7f66180cc170 2024-03-21T08:43:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.689+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f660801a070 con 0x7f66180cc170 2024-03-21T08:43:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.689+0000 7f660fe00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 0x7f65f4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:43:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.689+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6608086410 con 0x7f66180cc170 2024-03-21T08:43:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.689+0000 7f660e000640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 0x7f65f4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:43:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.693+0000 7f660e000640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 0x7f65f4069fe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f66000ae000 tx=0x7f6600008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:43:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.693+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f66080566f0 con 0x7f66180cc170 2024-03-21T08:43:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f661806d750 con 0x7f66180cc170 2024-03-21T08:43:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f660fe00640 1 -- 172.21.15.28:0/1233072299 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f660805a790 con 0x7f66180cc170 2024-03-21T08:43:59.864 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:43:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 msgr2=0x7f65f4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 0x7f65f4069fe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f66000ae000 tx=0x7f6600008040 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 msgr2=0x7f6618151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f660800e930 tx=0x7f660800edf0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 shutdown_connections 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f65f4067b20 0x7f65f4069fe0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f661815ba50 0x7f66180c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f66180ce320 0x7f6618159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 --2- 172.21.15.28:0/1233072299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66180cc170 0x7f6618151500 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 >> 172.21.15.28:0/1233072299 conn(0x7f661806fa90 msgr2=0x7f661806db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 shutdown_connections 2024-03-21T08:43:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:43:59.861+0000 7f6617e00640 1 -- 172.21.15.28:0/1233072299 wait complete. 2024-03-21T08:43:59.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:44:00.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.025+0000 7faadfa00640 1 Processor -- start 2024-03-21T08:44:00.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.025+0000 7faadfa00640 1 -- start start 2024-03-21T08:44:00.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadfa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadfa00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faad806ef80 0x7faad80cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadfa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad815d300 con 0x7faad81454c0 2024-03-21T08:44:00.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadfa00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad815c6f0 con 0x7faad806ef80 2024-03-21T08:44:00.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadfa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad815c9d0 con 0x7faad8072190 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadda00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faad806ef80 0x7faad80cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faacfe00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadd000640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7faad81454c0 0x7faad8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:41636/0 (socket says 172.21.15.28:41636) 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faadd000640 1 -- 172.21.15.28:0/3218980047 learned_addr learned my addr 172.21.15.28:0/3218980047 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faacfe00640 1 -- 172.21.15.28:0/3218980047 >> v2:172.21.15.151:3300/0 conn(0x7faad806ef80 msgr2=0x7faad80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faacfe00640 1 --2- 172.21.15.28:0/3218980047 >> v2:172.21.15.151:3300/0 conn(0x7faad806ef80 0x7faad80cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faacfe00640 1 -- 172.21.15.28:0/3218980047 >> v1:172.21.15.151:6789/0 conn(0x7faad81454c0 legacy=0x7faad8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faacfe00640 1 -- 172.21.15.28:0/3218980047 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad815ccb0 con 0x7faad8072190 2024-03-21T08:44:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.029+0000 7faad7e00640 1 -- 172.21.15.28:0/3218980047 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317819881 0 0) 0x7faad815d300 con 0x7faad81454c0 2024-03-21T08:44:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faacfe00640 1 --2- 172.21.15.28:0/3218980047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad815c0b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faac00097c0 tx=0x7faac001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58b389db839ff15 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faad7e00640 1 -- 172.21.15.28:0/3218980047 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac00aa5f0 con 0x7faad8072190 2024-03-21T08:44:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faad7e00640 1 -- 172.21.15.28:0/3218980047 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac00af070 con 0x7faad8072190 2024-03-21T08:44:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faad7e00640 1 -- 172.21.15.28:0/3218980047 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac00aa9b0 con 0x7faad8072190 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- 172.21.15.28:0/3218980047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 msgr2=0x7faad815c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 --2- 172.21.15.28:0/3218980047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad815c0b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faac00097c0 tx=0x7faac001c470 comp rx=0 tx=0).stop 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- 172.21.15.28:0/3218980047 shutdown_connections 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 --2- 172.21.15.28:0/3218980047 >> v2:172.21.15.151:3300/0 conn(0x7faad806ef80 0x7faad80cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 --2- 172.21.15.28:0/3218980047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad815c0b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- 172.21.15.28:0/3218980047 >> 172.21.15.28:0/3218980047 conn(0x7faad80674f0 msgr2=0x7faad806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- 172.21.15.28:0/3218980047 shutdown_connections 2024-03-21T08:44:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- 172.21.15.28:0/3218980047 wait complete. 2024-03-21T08:44:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 Processor -- start 2024-03-21T08:44:00.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.033+0000 7faadfa00640 1 -- start start 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad81346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 0x7faad81411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad80cd5a0 con 0x7faad806ef80 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad80cd420 con 0x7faad81454c0 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad80cd720 con 0x7faad8072190 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53256/0 (socket says 172.21.15.28:53256) 2024-03-21T08:44:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 -- 172.21.15.28:0/89351556 learned_addr learned my addr 172.21.15.28:0/89351556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadda00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 0x7faad81411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 -- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 msgr2=0x7faad81346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faacfe00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad81346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad81346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 -- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 msgr2=0x7faad81411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 0x7faad81411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 -- 172.21.15.28:0/89351556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8134c20 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadda00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 0x7faad81411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faacfe00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad81346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadd000640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faac800c480 tx=0x7faac800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8018070 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac8007500 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8004a30 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8134ec0 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8135480 con 0x7faad806ef80 2024-03-21T08:44:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.037+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad8073430 con 0x7faad806ef80 2024-03-21T08:44:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.041+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faac8004bd0 con 0x7faad806ef80 2024-03-21T08:44:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.041+0000 7faad5600640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 0x7faab406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.041+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faac80852f0 con 0x7faad806ef80 2024-03-21T08:44:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.041+0000 7faacfe00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 0x7faab406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.041+0000 7faacfe00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 0x7faab406a010 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faac00ae000 tx=0x7faac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.049+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faac8055b60 con 0x7faad806ef80 2024-03-21T08:44:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faad80ae990 con 0x7faad806ef80 2024-03-21T08:44:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faad5600640 1 -- 172.21.15.28:0/89351556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7faac8055460 con 0x7faad806ef80 2024-03-21T08:44:00.230 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:44:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 msgr2=0x7faab406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 0x7faab406a010 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faac00ae000 tx=0x7faac0008040 comp rx=0 tx=0).stop 2024-03-21T08:44:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 msgr2=0x7faad815bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faac800c480 tx=0x7faac800c940 comp rx=0 tx=0).stop 2024-03-21T08:44:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 shutdown_connections 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faad81454c0 0x7faad81411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faab4067b50 0x7faab406a010 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faad8072190 0x7faad81346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 --2- 172.21.15.28:0/89351556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad806ef80 0x7faad815bf00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 >> 172.21.15.28:0/89351556 conn(0x7faad80674f0 msgr2=0x7faad81446e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 shutdown_connections 2024-03-21T08:44:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.225+0000 7faadfa00640 1 -- 172.21.15.28:0/89351556 wait complete. 2024-03-21T08:44:00.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:44:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 Processor -- start 2024-03-21T08:44:00.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 -- start start 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 0x7f6e4408afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4415fba0 con 0x7f6e440b0e80 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4415fe80 con 0x7f6e440cebd0 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e4b000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44160160 con 0x7f6e4415bd20 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e3b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6e440b0e80 0x7f6e440ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43514/0 (socket says 172.21.15.28:43514) 2024-03-21T08:44:00.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.389+0000 7f6e3b400640 1 -- 172.21.15.28:0/1452623639 learned_addr learned my addr 172.21.15.28:0/1452623639 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 --2- 172.21.15.28:0/1452623639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e49000640 1 --2- 172.21.15.28:0/1452623639 >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 0x7f6e4408afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43400640 1 -- 172.21.15.28:0/1452623639 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199121199 0 0) 0x7f6e4415fba0 con 0x7f6e440b0e80 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43400640 1 -- 172.21.15.28:0/1452623639 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e28003530 con 0x7f6e440b0e80 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 -- 172.21.15.28:0/1452623639 >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 msgr2=0x7f6e4408afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 --2- 172.21.15.28:0/1452623639 >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 0x7f6e4408afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 -- 172.21.15.28:0/1452623639 >> v1:172.21.15.28:6789/0 conn(0x7f6e440b0e80 legacy=0x7f6e440ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 -- 172.21.15.28:0/1452623639 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44160440 con 0x7f6e4415bd20 2024-03-21T08:44:00.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e49000640 1 --2- 172.21.15.28:0/1452623639 >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 0x7f6e4408afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43e00640 1 --2- 172.21.15.28:0/1452623639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440b2c40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e340097c0 tx=0x7f6e3401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19be33e1c5880163 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43400640 1 -- 172.21.15.28:0/1452623639 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e340b0070 con 0x7f6e4415bd20 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43400640 1 -- 172.21.15.28:0/1452623639 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e340ab600 con 0x7f6e4415bd20 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.393+0000 7f6e43400640 1 -- 172.21.15.28:0/1452623639 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e340ab9d0 con 0x7f6e4415bd20 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- 172.21.15.28:0/1452623639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 msgr2=0x7f6e440b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- 172.21.15.28:0/1452623639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440b2c40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6e340097c0 tx=0x7f6e3401c4a0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- 172.21.15.28:0/1452623639 shutdown_connections 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- 172.21.15.28:0/1452623639 >> v2:172.21.15.151:3300/0 conn(0x7f6e440cebd0 0x7f6e4408afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- 172.21.15.28:0/1452623639 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440b2c40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- 172.21.15.28:0/1452623639 >> 172.21.15.28:0/1452623639 conn(0x7f6e440d1ab0 msgr2=0x7f6e440d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- 172.21.15.28:0/1452623639 shutdown_connections 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- 172.21.15.28:0/1452623639 wait complete. 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 Processor -- start 2024-03-21T08:44:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- start start 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e440b0e80 0x7f6e4415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4408be90 con 0x7f6e440cebd0 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4408bd10 con 0x7f6e440b0e80 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e4b000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44168d10 con 0x7f6e4415bd20 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52516/0 (socket says 172.21.15.28:52516) 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e3b400640 1 -- 172.21.15.28:0/180619958 learned_addr learned my addr 172.21.15.28:0/180619958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e49000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.397+0000 7f6e43e00640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e440b0e80 0x7f6e4415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 -- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 msgr2=0x7f6e440cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 -- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e440b0e80 msgr2=0x7f6e4415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e440b0e80 0x7f6e4415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 -- 172.21.15.28:0/180619958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4414bd70 con 0x7f6e440cebd0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e3b400640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00c450 tx=0x7f6e2c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c017070 con 0x7f6e440cebd0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2c005cf0 con 0x7f6e440cebd0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c0072f0 con 0x7f6e440cebd0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4414c0b0 con 0x7f6e440cebd0 2024-03-21T08:44:00.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4414c8f0 con 0x7f6e440cebd0 2024-03-21T08:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4404f4f0 con 0x7f6e440cebd0 2024-03-21T08:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.401+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6e2c01a070 con 0x7f6e440cebd0 2024-03-21T08:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.405+0000 7f6e40c00640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 0x7f6e20069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.405+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e2c0853c0 con 0x7f6e440cebd0 2024-03-21T08:44:00.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.405+0000 7f6e43e00640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 0x7f6e20069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.405+0000 7f6e43e00640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 0x7f6e20069fe0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ad000 tx=0x7f6e340a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.409+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6e4414c0b0 con 0x7f6e440cebd0 2024-03-21T08:44:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.569+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e440d2830 con 0x7f6e440cebd0 2024-03-21T08:44:00.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.569+0000 7f6e40c00640 1 -- 172.21.15.28:0/180619958 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e2c0555f0 con 0x7f6e440cebd0 2024-03-21T08:44:00.573 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 msgr2=0x7f6e20069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 0x7f6e20069fe0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ad000 tx=0x7f6e340a6000 comp rx=0 tx=0).stop 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 msgr2=0x7f6e440cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00c450 tx=0x7f6e2c00c850 comp rx=0 tx=0).stop 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 shutdown_connections 2024-03-21T08:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6e20067b20 0x7f6e20069fe0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6e4415bd20 0x7f6e440cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e440cebd0 0x7f6e440cb0f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 --2- 172.21.15.28:0/180619958 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6e440b0e80 0x7f6e4415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 >> 172.21.15.28:0/180619958 conn(0x7f6e440d1ab0 msgr2=0x7f6e440b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 shutdown_connections 2024-03-21T08:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.573+0000 7f6e4b000640 1 -- 172.21.15.28:0/180619958 wait complete. 2024-03-21T08:44:00.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:44:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.733+0000 7f9ebd000640 1 Processor -- start 2024-03-21T08:44:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.733+0000 7f9ebd000640 1 -- start start 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9eb815e2a0 0x7f9eb809aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb8099c90 0x7f9eb809a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9eb806c520 0x7f9eb806c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb809a630 con 0x7f9eb815e2a0 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb80d0930 con 0x7f9eb806c520 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9ebd000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80d0c10 con 0x7f9eb8099c90 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9eb815e2a0 0x7f9eb809aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9eb815e2a0 0x7f9eb809aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52520/0 (socket says 172.21.15.28:52520) 2024-03-21T08:44:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 -- 172.21.15.28:0/2589875655 learned_addr learned my addr 172.21.15.28:0/2589875655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb6800640 1 --2- 172.21.15.28:0/2589875655 >> v2:172.21.15.151:3300/0 conn(0x7f9eb806c520 0x7f9eb806c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eafe00640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb8099c90 0x7f9eb809a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 -- 172.21.15.28:0/2589875655 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb8099c90 msgr2=0x7f9eb809a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb8099c90 0x7f9eb809a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 -- 172.21.15.28:0/2589875655 >> v2:172.21.15.151:3300/0 conn(0x7f9eb806c520 msgr2=0x7f9eb806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2589875655 >> v2:172.21.15.151:3300/0 conn(0x7f9eb806c520 0x7f9eb806c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.737+0000 7f9eb5e00640 1 -- 172.21.15.28:0/2589875655 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb80d0ef0 con 0x7f9eb815e2a0 2024-03-21T08:44:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb815e2a0 0x7f9eb809aad0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80c5630 tx=0x7f9ea801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3942e54bc8da8f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9eb5400640 1 -- 172.21.15.28:0/2589875655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea80b2070 con 0x7f9eb815e2a0 2024-03-21T08:44:00.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9eb5400640 1 -- 172.21.15.28:0/2589875655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea80a9070 con 0x7f9eb815e2a0 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9eb5400640 1 -- 172.21.15.28:0/2589875655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea80ad880 con 0x7f9eb815e2a0 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- 172.21.15.28:0/2589875655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb815e2a0 msgr2=0x7f9eb809aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb815e2a0 0x7f9eb809aad0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80c5630 tx=0x7f9ea801c490 comp rx=0 tx=0).stop 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- 172.21.15.28:0/2589875655 shutdown_connections 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2589875655 >> v2:172.21.15.151:3300/0 conn(0x7f9eb806c520 0x7f9eb806c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb8099c90 0x7f9eb809a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2589875655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb815e2a0 0x7f9eb809aad0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- 172.21.15.28:0/2589875655 >> 172.21.15.28:0/2589875655 conn(0x7f9eb80b3620 msgr2=0x7f9eb80b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- 172.21.15.28:0/2589875655 shutdown_connections 2024-03-21T08:44:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- 172.21.15.28:0/2589875655 wait complete. 2024-03-21T08:44:00.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 Processor -- start 2024-03-21T08:44:00.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- start start 2024-03-21T08:44:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9eb806c520 0x7f9eb80b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb815e2a0 0x7f9eb80cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb815e680 con 0x7f9eb8099c90 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb809b010 con 0x7f9eb806c520 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.741+0000 7f9ebd000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb815ec90 con 0x7f9eb815e2a0 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52524/0 (socket says 172.21.15.28:52524) 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 -- 172.21.15.28:0/2545400559 learned_addr learned my addr 172.21.15.28:0/2545400559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9eb806c520 0x7f9eb80b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eb6800640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb815e2a0 0x7f9eb80cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 -- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb815e2a0 msgr2=0x7f9eb80cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb815e2a0 0x7f9eb80cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 -- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9eb806c520 msgr2=0x7f9eb80b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9eb806c520 0x7f9eb80b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 -- 172.21.15.28:0/2545400559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8154530 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eafe00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000d830 tx=0x7f9ea000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000b6c0 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea0019070 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb8153f70 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000ba60 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb8160420 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb809faf0 con 0x7f9eb8099c90 2024-03-21T08:44:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9ea001e460 con 0x7f9eb8099c90 2024-03-21T08:44:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.745+0000 7f9eae000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 0x7f9e9406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.749+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ea00866f0 con 0x7f9eb8099c90 2024-03-21T08:44:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.749+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 0x7f9e9406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.749+0000 7f9eb5e00640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 0x7f9e9406a010 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80af000 tx=0x7f9ea80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:00.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.753+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9ea00567f0 con 0x7f9eb8099c90 2024-03-21T08:44:00.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.929+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9eb804f4f0 con 0x7f9eb8099c90 2024-03-21T08:44:00.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.929+0000 7f9eae000640 1 -- 172.21.15.28:0/2545400559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9ea005a890 con 0x7f9eb8099c90 2024-03-21T08:44:00.933 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:44:00.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 msgr2=0x7f9e9406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 0x7f9e9406a010 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80af000 tx=0x7f9ea80a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 msgr2=0x7f9eb815d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000d830 tx=0x7f9ea000dcf0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 shutdown_connections 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9e94067b50 0x7f9e9406a010 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9eb815e2a0 0x7f9eb80cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb8099c90 0x7f9eb815d4d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 --2- 172.21.15.28:0/2545400559 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9eb806c520 0x7f9eb80b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 >> 172.21.15.28:0/2545400559 conn(0x7f9eb80b3620 msgr2=0x7f9eb806d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 shutdown_connections 2024-03-21T08:44:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:00.933+0000 7f9ebd000640 1 -- 172.21.15.28:0/2545400559 wait complete. 2024-03-21T08:44:00.955 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:44:00.955 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:00.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:01.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 Processor -- start 2024-03-21T08:44:01.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 -- start start 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f9d80099b80 0x7f9d80099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 0x7f9d800c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d800cdb50 con 0x7f9d8015f260 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8009a520 con 0x7f9d80099b80 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d85600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800c5530 con 0x7f9d800c5f80 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9d8015f260 0x7f9d800cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43518/0 (socket says 172.21.15.28:43518) 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7ea00640 1 -- 172.21.15.28:0/2945083073 learned_addr learned my addr 172.21.15.28:0/2945083073 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7f400640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 0x7f9d800c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d77e00640 1 --2- 172.21.15.28:0/2945083073 >> v2:172.21.15.151:3300/0 conn(0x7f9d80099b80 0x7f9d80099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7e000640 1 -- 172.21.15.28:0/2945083073 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601263639 0 0) 0x7f9d800cdb50 con 0x7f9d8015f260 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7e000640 1 -- 172.21.15.28:0/2945083073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d64003530 con 0x7f9d8015f260 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d77e00640 1 -- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 msgr2=0x7f9d800c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d77e00640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 0x7f9d800c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d77e00640 1 -- 172.21.15.28:0/2945083073 >> v1:172.21.15.28:6789/0 conn(0x7f9d8015f260 legacy=0x7f9d800cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d77e00640 1 -- 172.21.15.28:0/2945083073 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d800cde30 con 0x7f9d80099b80 2024-03-21T08:44:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.153+0000 7f9d7f400640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 0x7f9d800c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d77e00640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d80099b80 0x7f9d80099fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9d680097c0 tx=0x7f9d6801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=121a9d72da43b077 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d7e000640 1 -- 172.21.15.28:0/2945083073 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680b1070 con 0x7f9d80099b80 2024-03-21T08:44:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d7e000640 1 -- 172.21.15.28:0/2945083073 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d680a8050 con 0x7f9d80099b80 2024-03-21T08:44:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d7e000640 1 -- 172.21.15.28:0/2945083073 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680ac860 con 0x7f9d80099b80 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 -- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d80099b80 msgr2=0x7f9d80099fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d80099b80 0x7f9d80099fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9d680097c0 tx=0x7f9d6801c3e0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 -- 172.21.15.28:0/2945083073 shutdown_connections 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d800c5f80 0x7f9d800c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 --2- 172.21.15.28:0/2945083073 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d80099b80 0x7f9d80099fe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 -- 172.21.15.28:0/2945083073 >> 172.21.15.28:0/2945083073 conn(0x7f9d800b34d0 msgr2=0x7f9d800b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 -- 172.21.15.28:0/2945083073 shutdown_connections 2024-03-21T08:44:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.157+0000 7f9d85600640 1 -- 172.21.15.28:0/2945083073 wait complete. 2024-03-21T08:44:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 Processor -- start 2024-03-21T08:44:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- start start 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 0x7f9d80156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 0x7f9d8015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800cd7a0 con 0x7f9d800c5f80 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8015f640 con 0x7f9d8015f260 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80159c20 con 0x7f9d80099b80 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52540/0 (socket says 172.21.15.28:52540) 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 -- 172.21.15.28:0/4291719433 learned_addr learned my addr 172.21.15.28:0/4291719433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d7f400640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 0x7f9d8015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d7ea00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 0x7f9d80156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 -- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 msgr2=0x7f9d80156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 0x7f9d80156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 -- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 msgr2=0x7f9d8015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 0x7f9d8015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 -- 172.21.15.28:0/4291719433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80159ec0 con 0x7f9d800c5f80 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d7f400640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 0x7f9d8015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d7ea00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 0x7f9d80156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d77e00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9d6801aa90 tx=0x7f9d6801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680b1040 con 0x7f9d800c5f80 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d680bc070 con 0x7f9d800c5f80 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d680b7450 con 0x7f9d800c5f80 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8015b880 con 0x7f9d800c5f80 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8015a1a0 con 0x7f9d800c5f80 2024-03-21T08:44:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.161+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8015b670 con 0x7f9d800c5f80 2024-03-21T08:44:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.165+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f9d800c7160 con 0x7f9d800c5f80 2024-03-21T08:44:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.165+0000 7f9d76a00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 0x7f9d5c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.165+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d6811ffe0 con 0x7f9d800c5f80 2024-03-21T08:44:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.165+0000 7f9d7ea00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 0x7f9d5c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.165+0000 7f9d7ea00640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 0x7f9d5c06e420 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9d70005890 tx=0x7f9d70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.169+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9d68016090 con 0x7f9d800c5f80 2024-03-21T08:44:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d8004f410 con 0x7f9d800c5f80 2024-03-21T08:44:01.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d76a00640 1 -- 172.21.15.28:0/4291719433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f9d680b4380 con 0x7f9d800c5f80 2024-03-21T08:44:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:01.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 msgr2=0x7f9d5c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 0x7f9d5c06e420 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9d70005890 tx=0x7f9d70009040 comp rx=0 tx=0).stop 2024-03-21T08:44:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 msgr2=0x7f9d800b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9d6801aa90 tx=0x7f9d6801ac20 comp rx=0 tx=0).stop 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 shutdown_connections 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f9d8015f260 0x7f9d8015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d800c5f80 0x7f9d800b6e10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9d5c06bf60 0x7f9d5c06e420 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 --2- 172.21.15.28:0/4291719433 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f9d80099b80 0x7f9d80156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 >> 172.21.15.28:0/4291719433 conn(0x7f9d800b34d0 msgr2=0x7f9d8006fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 shutdown_connections 2024-03-21T08:44:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.333+0000 7f9d85600640 1 -- 172.21.15.28:0/4291719433 wait complete. 2024-03-21T08:44:01.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:44:01.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:44:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 Processor -- start 2024-03-21T08:44:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 -- start start 2024-03-21T08:44:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68180d0360 0x7f681806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 0x7f68180cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b3010 con 0x7f681815e1e0 2024-03-21T08:44:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681806d310 con 0x7f68180d0360 2024-03-21T08:44:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681f400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681806d5f0 con 0x7f681806cbe0 2024-03-21T08:44:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68180d0360 0x7f681806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68180d0360 0x7f681806c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52550/0 (socket says 172.21.15.28:52550) 2024-03-21T08:44:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 -- 172.21.15.28:0/678005428 learned_addr learned my addr 172.21.15.28:0/678005428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681d400640 1 --2- 172.21.15.28:0/678005428 >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 0x7f68180cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 -- 172.21.15.28:0/678005428 >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 msgr2=0x7f68180cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 --2- 172.21.15.28:0/678005428 >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 0x7f68180cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 -- 172.21.15.28:0/678005428 >> v1:172.21.15.151:6789/0 conn(0x7f681815e1e0 legacy=0x7f68180cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f6817e00640 1 -- 172.21.15.28:0/678005428 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681806d8d0 con 0x7f68180d0360 2024-03-21T08:44:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.501+0000 7f681d400640 1 --2- 172.21.15.28:0/678005428 >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 0x7f68180cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f6817e00640 1 --2- 172.21.15.28:0/678005428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68180d0360 0x7f681806c6a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f681806e380 tx=0x7f680401cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d15578d13f9795c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f6817400640 1 -- 172.21.15.28:0/678005428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68040ab430 con 0x7f68180d0360 2024-03-21T08:44:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f6817400640 1 -- 172.21.15.28:0/678005428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68040b0070 con 0x7f68180d0360 2024-03-21T08:44:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f6817400640 1 -- 172.21.15.28:0/678005428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68040ab7f0 con 0x7f68180d0360 2024-03-21T08:44:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f681f400640 1 -- 172.21.15.28:0/678005428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68180d0360 msgr2=0x7f681806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.505+0000 7f681f400640 1 --2- 172.21.15.28:0/678005428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68180d0360 0x7f681806c6a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f681806e380 tx=0x7f680401cf80 comp rx=0 tx=0).stop 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- 172.21.15.28:0/678005428 shutdown_connections 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 --2- 172.21.15.28:0/678005428 >> v2:172.21.15.151:3300/0 conn(0x7f681806cbe0 0x7f68180cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 --2- 172.21.15.28:0/678005428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68180d0360 0x7f681806c6a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- 172.21.15.28:0/678005428 >> 172.21.15.28:0/678005428 conn(0x7f68180d1ab0 msgr2=0x7f68180d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- 172.21.15.28:0/678005428 shutdown_connections 2024-03-21T08:44:01.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- 172.21.15.28:0/678005428 wait complete. 2024-03-21T08:44:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 Processor -- start 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- start start 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68180d0360 0x7f6818149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f681815e1e0 0x7f68180b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cd600 con 0x7f681806cbe0 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cd480 con 0x7f68180d0360 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681f400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180cd780 con 0x7f681815e1e0 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52558/0 (socket says 172.21.15.28:52558) 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 -- 172.21.15.28:0/3354933956 learned_addr learned my addr 172.21.15.28:0/3354933956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681d400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f681815e1e0 0x7f68180b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f6817e00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68180d0360 0x7f6818149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 -- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f681815e1e0 msgr2=0x7f68180b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f681815e1e0 0x7f68180b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 -- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68180d0360 msgr2=0x7f6818149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68180d0360 0x7f6818149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 -- 172.21.15.28:0/3354933956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681814e260 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.509+0000 7f681ca00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c420 tx=0x7f680c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0052d0 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c005470 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681814e540 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c007260 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681814eb00 con 0x7f681806cbe0 2024-03-21T08:44:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681815da10 con 0x7f681806cbe0 2024-03-21T08:44:01.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.513+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f680c007400 con 0x7f681806cbe0 2024-03-21T08:44:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.517+0000 7f6814c00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 0x7f67f806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.517+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f680c085550 con 0x7f681806cbe0 2024-03-21T08:44:01.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.517+0000 7f6817e00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 0x7f67f806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.517+0000 7f6817e00640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 0x7f67f806a010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f68040af000 tx=0x7f6804004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.521+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f680c055650 con 0x7f681806cbe0 2024-03-21T08:44:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68180d2830 con 0x7f681806cbe0 2024-03-21T08:44:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f6814c00640 1 -- 172.21.15.28:0/3354933956 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f680c0596f0 con 0x7f681806cbe0 2024-03-21T08:44:01.688 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 msgr2=0x7f67f806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 0x7f67f806a010 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f68040af000 tx=0x7f6804004740 comp rx=0 tx=0).stop 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 msgr2=0x7f68181501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f680c00c420 tx=0x7f680c00c820 comp rx=0 tx=0).stop 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 shutdown_connections 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f67f8067b50 0x7f67f806a010 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f681815e1e0 0x7f68180b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f68180d0360 0x7f6818149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 --2- 172.21.15.28:0/3354933956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681806cbe0 0x7f68181501b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 >> 172.21.15.28:0/3354933956 conn(0x7f68180d1ab0 msgr2=0x7f68180c9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 shutdown_connections 2024-03-21T08:44:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.685+0000 7f681f400640 1 -- 172.21.15.28:0/3354933956 wait complete. 2024-03-21T08:44:01.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:44:01.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 Processor -- start 2024-03-21T08:44:01.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 -- start start 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f199815dd40 0x7f19980cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f19981626d0 0x7f19980ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199808b8f0 con 0x7f199815e120 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199808b6d0 con 0x7f199815dd40 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.877+0000 7f199fc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199806c280 con 0x7f19981626d0 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f199815dd40 0x7f19980cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f199815e120 0x7f199808afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43524/0 (socket says 172.21.15.28:43524) 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997e00640 1 -- 172.21.15.28:0/2673821352 learned_addr learned my addr 172.21.15.28:0/2673821352 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199dc00640 1 --2- 172.21.15.28:0/2673821352 >> v2:172.21.15.151:3300/0 conn(0x7f19981626d0 0x7f19980ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 -- 172.21.15.28:0/2673821352 >> v2:172.21.15.151:3300/0 conn(0x7f19981626d0 msgr2=0x7f19980ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 --2- 172.21.15.28:0/2673821352 >> v2:172.21.15.151:3300/0 conn(0x7f19981626d0 0x7f19980ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 -- 172.21.15.28:0/2673821352 >> v1:172.21.15.28:6789/0 conn(0x7f199815e120 legacy=0x7f199808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 -- 172.21.15.28:0/2673821352 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199808bbd0 con 0x7f199815dd40 2024-03-21T08:44:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997400640 1 -- 172.21.15.28:0/2673821352 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261755668 0 0) 0x7f199808b8f0 con 0x7f199815e120 2024-03-21T08:44:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f199d200640 1 --2- 172.21.15.28:0/2673821352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199815dd40 0x7f19980cfd60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f19980cfcb0 tx=0x7f198c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21dea53e38eb23f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997400640 1 -- 172.21.15.28:0/2673821352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0b0070 con 0x7f199815dd40 2024-03-21T08:44:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997400640 1 -- 172.21.15.28:0/2673821352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0ab600 con 0x7f199815dd40 2024-03-21T08:44:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.881+0000 7f1997400640 1 -- 172.21.15.28:0/2673821352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0ab9d0 con 0x7f199815dd40 2024-03-21T08:44:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- 172.21.15.28:0/2673821352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199815dd40 msgr2=0x7f19980cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- 172.21.15.28:0/2673821352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199815dd40 0x7f19980cfd60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f19980cfcb0 tx=0x7f198c01c3f0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- 172.21.15.28:0/2673821352 shutdown_connections 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- 172.21.15.28:0/2673821352 >> v2:172.21.15.151:3300/0 conn(0x7f19981626d0 0x7f19980ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- 172.21.15.28:0/2673821352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199815dd40 0x7f19980cfd60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- 172.21.15.28:0/2673821352 >> 172.21.15.28:0/2673821352 conn(0x7f19980d1b20 msgr2=0x7f19980d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- 172.21.15.28:0/2673821352 shutdown_connections 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- 172.21.15.28:0/2673821352 wait complete. 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 Processor -- start 2024-03-21T08:44:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- start start 2024-03-21T08:44:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 0x7f19980b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 0x7f199815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199806d590 con 0x7f19981626d0 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199806d410 con 0x7f199815e120 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199816ad60 con 0x7f199815dd40 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52564/0 (socket says 172.21.15.28:52564) 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 -- 172.21.15.28:0/220450989 learned_addr learned my addr 172.21.15.28:0/220450989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f1997e00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 0x7f199815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199d200640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 0x7f19980b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 -- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 msgr2=0x7f19980b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 0x7f19980b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 -- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 msgr2=0x7f199815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 0x7f199815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199dc00640 1 -- 172.21.15.28:0/220450989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19981545f0 con 0x7f19981626d0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f199d200640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 0x7f19980b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.885+0000 7f1997e00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 0x7f199815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f199dc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f199000efc0 tx=0x7f199000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990018020 con 0x7f19981626d0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1990014480 con 0x7f19981626d0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990009e60 con 0x7f19981626d0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1998153270 con 0x7f19981626d0 2024-03-21T08:44:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1998153830 con 0x7f19981626d0 2024-03-21T08:44:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19981542d0 con 0x7f19981626d0 2024-03-21T08:44:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f199001b020 con 0x7f19981626d0 2024-03-21T08:44:01.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 0x7f197806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:01.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1990086df0 con 0x7f19981626d0 2024-03-21T08:44:01.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.889+0000 7f199d200640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 0x7f197806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:01.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.893+0000 7f199d200640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 0x7f197806a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f198c0ad000 tx=0x7f198c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:01.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:01.901+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f19981542d0 con 0x7f19981626d0 2024-03-21T08:44:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f199804f4f0 con 0x7f19981626d0 2024-03-21T08:44:02.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f1994c00640 1 -- 172.21.15.28:0/220450989 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1990022030 con 0x7f19981626d0 2024-03-21T08:44:02.064 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:44:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 msgr2=0x7f197806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 0x7f197806a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f198c0ad000 tx=0x7f198c008040 comp rx=0 tx=0).stop 2024-03-21T08:44:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 msgr2=0x7f1998155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f199000efc0 tx=0x7f199000c490 comp rx=0 tx=0).stop 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 shutdown_connections 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19981626d0 0x7f1998155040 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f199815e120 0x7f199815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1978067b50 0x7f197806a010 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 --2- 172.21.15.28:0/220450989 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f199815dd40 0x7f19980b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.061+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 >> 172.21.15.28:0/220450989 conn(0x7f19980d1b20 msgr2=0x7f199806fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.065+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 shutdown_connections 2024-03-21T08:44:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.065+0000 7f199fc00640 1 -- 172.21.15.28:0/220450989 wait complete. 2024-03-21T08:44:02.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:44:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.229+0000 7fd157a00640 1 Processor -- start 2024-03-21T08:44:02.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.229+0000 7fd157a00640 1 -- start start 2024-03-21T08:44:02.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd157a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd15015bd20 0x7fd1500b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd157a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1500cebd0 0x7fd15008afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd157a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15015fba0 con 0x7fd1500b0e80 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd157a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15015fe80 con 0x7fd1500cebd0 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd157a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd150160160 con 0x7fd15015bd20 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd14fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1500b0e80 0x7fd1500ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43532/0 (socket says 172.21.15.28:43532) 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd14fe00640 1 -- 172.21.15.28:0/1958937965 learned_addr learned my addr 172.21.15.28:0/1958937965 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155a00640 1 --2- 172.21.15.28:0/1958937965 >> v2:172.21.15.28:3300/0 conn(0x7fd1500cebd0 0x7fd15008afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155000640 1 --2- 172.21.15.28:0/1958937965 >> v2:172.21.15.151:3300/0 conn(0x7fd15015bd20 0x7fd1500b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd14f400640 1 -- 172.21.15.28:0/1958937965 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941179440 0 0) 0x7fd15015fba0 con 0x7fd1500b0e80 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd14f400640 1 -- 172.21.15.28:0/1958937965 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd138003530 con 0x7fd1500b0e80 2024-03-21T08:44:02.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155a00640 1 -- 172.21.15.28:0/1958937965 >> v2:172.21.15.151:3300/0 conn(0x7fd15015bd20 msgr2=0x7fd1500b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:02.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155a00640 1 --2- 172.21.15.28:0/1958937965 >> v2:172.21.15.151:3300/0 conn(0x7fd15015bd20 0x7fd1500b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155a00640 1 -- 172.21.15.28:0/1958937965 >> v1:172.21.15.28:6789/0 conn(0x7fd1500b0e80 legacy=0x7fd1500ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.233+0000 7fd155a00640 1 -- 172.21.15.28:0/1958937965 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150160440 con 0x7fd1500cebd0 2024-03-21T08:44:02.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd155a00640 1 --2- 172.21.15.28:0/1958937965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd15008afc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd150068b20 tx=0x7fd14801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b7c0ae5ab9716b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd14f400640 1 -- 172.21.15.28:0/1958937965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1480b2070 con 0x7fd1500cebd0 2024-03-21T08:44:02.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd14f400640 1 -- 172.21.15.28:0/1958937965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1480a57f0 con 0x7fd1500cebd0 2024-03-21T08:44:02.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd14f400640 1 -- 172.21.15.28:0/1958937965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1480a5bc0 con 0x7fd1500cebd0 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 -- 172.21.15.28:0/1958937965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 msgr2=0x7fd15008afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 --2- 172.21.15.28:0/1958937965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd15008afc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd150068b20 tx=0x7fd14801c930 comp rx=0 tx=0).stop 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 -- 172.21.15.28:0/1958937965 shutdown_connections 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 --2- 172.21.15.28:0/1958937965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd15008afc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 --2- 172.21.15.28:0/1958937965 >> v2:172.21.15.151:3300/0 conn(0x7fd15015bd20 0x7fd1500b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 -- 172.21.15.28:0/1958937965 >> 172.21.15.28:0/1958937965 conn(0x7fd1500d1ab0 msgr2=0x7fd1500d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 -- 172.21.15.28:0/1958937965 shutdown_connections 2024-03-21T08:44:02.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.237+0000 7fd157a00640 1 -- 172.21.15.28:0/1958937965 wait complete. 2024-03-21T08:44:02.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 Processor -- start 2024-03-21T08:44:02.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- start start 2024-03-21T08:44:02.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 0x7fd150154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd15015bd20 0x7fd1501415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15008be90 con 0x7fd1500cebd0 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15008bd10 con 0x7fd15015bd20 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd150168d10 con 0x7fd1500b0e80 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52584/0 (socket says 172.21.15.28:52584) 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 -- 172.21.15.28:0/1867941184 learned_addr learned my addr 172.21.15.28:0/1867941184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd155000640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 0x7fd150154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd155a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd15015bd20 0x7fd1501415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 -- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 msgr2=0x7fd150154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 0x7fd150154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 -- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd15015bd20 msgr2=0x7fd1501415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd15015bd20 0x7fd1501415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 -- 172.21.15.28:0/1867941184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15015e4e0 con 0x7fd1500cebd0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd155000640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 0x7fd150154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14fe00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00c3c0 tx=0x7fd13c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c017070 con 0x7fd1500cebd0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c005cf0 con 0x7fd1500cebd0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15015e7c0 con 0x7fd1500cebd0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0072f0 con 0x7fd1500cebd0 2024-03-21T08:44:02.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1500b72e0 con 0x7fd1500cebd0 2024-03-21T08:44:02.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.241+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15004f4f0 con 0x7fd1500cebd0 2024-03-21T08:44:02.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.245+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd13c01a070 con 0x7fd1500cebd0 2024-03-21T08:44:02.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.245+0000 7fd14cc00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 0x7fd130069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.245+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd13c085510 con 0x7fd1500cebd0 2024-03-21T08:44:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.245+0000 7fd155000640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 0x7fd130069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.245+0000 7fd155000640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 0x7fd130069fe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd150155770 tx=0x7fd144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.249+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd13c0556a0 con 0x7fd1500cebd0 2024-03-21T08:44:02.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.409+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1500d2830 con 0x7fd1500cebd0 2024-03-21T08:44:02.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.409+0000 7fd14cc00640 1 -- 172.21.15.28:0/1867941184 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd13c059740 con 0x7fd1500cebd0 2024-03-21T08:44:02.414 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 msgr2=0x7fd130069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 0x7fd130069fe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd150155770 tx=0x7fd144008040 comp rx=0 tx=0).stop 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 msgr2=0x7fd1500b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00c3c0 tx=0x7fd13c00c880 comp rx=0 tx=0).stop 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 shutdown_connections 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd15015bd20 0x7fd1501415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1500cebd0 0x7fd1500b6da0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd130067b20 0x7fd130069fe0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 --2- 172.21.15.28:0/1867941184 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd1500b0e80 0x7fd150154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 >> 172.21.15.28:0/1867941184 conn(0x7fd1500d1ab0 msgr2=0x7fd15006fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 shutdown_connections 2024-03-21T08:44:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.413+0000 7fd157a00640 1 -- 172.21.15.28:0/1867941184 wait complete. 2024-03-21T08:44:02.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:44:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.601+0000 7ff97b400640 1 Processor -- start 2024-03-21T08:44:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.601+0000 7ff97b400640 1 -- start start 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff97b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff974071a80 0x7ff97406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff97b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740ce5b0 con 0x7ff9740cde40 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff97b400640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740ce890 con 0x7ff97415cbb0 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff97b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97415c720 con 0x7ff974071a80 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff974071a80 0x7ff97406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff974071a80 0x7ff97406f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52596/0 (socket says 172.21.15.28:52596) 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 -- 172.21.15.28:0/943845107 learned_addr learned my addr 172.21.15.28:0/943845107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff96be00640 1 --1- 172.21.15.28:0/943845107 >> v1:172.21.15.28:6789/0 conn(0x7ff9740cde40 0x7ff974071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43542/0 (socket says 172.21.15.28:43542) 2024-03-21T08:44:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 -- 172.21.15.28:0/943845107 >> v1:172.21.15.151:6789/0 conn(0x7ff97415cbb0 legacy=0x7ff9740cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 -- 172.21.15.28:0/943845107 >> v1:172.21.15.28:6789/0 conn(0x7ff9740cde40 legacy=0x7ff974071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.605+0000 7ff978a00640 1 -- 172.21.15.28:0/943845107 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97415c3b0 con 0x7ff974071a80 2024-03-21T08:44:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff978a00640 1 --2- 172.21.15.28:0/943845107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff974071a80 0x7ff97406f0a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff964009870 tx=0x7ff96401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab895f0e615ca2d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff973e00640 1 -- 172.21.15.28:0/943845107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9640aa610 con 0x7ff974071a80 2024-03-21T08:44:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff973e00640 1 -- 172.21.15.28:0/943845107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9640af070 con 0x7ff974071a80 2024-03-21T08:44:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff973e00640 1 -- 172.21.15.28:0/943845107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9640aa9d0 con 0x7ff974071a80 2024-03-21T08:44:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 -- 172.21.15.28:0/943845107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff974071a80 msgr2=0x7ff97406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 --2- 172.21.15.28:0/943845107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff974071a80 0x7ff97406f0a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff964009870 tx=0x7ff96401c490 comp rx=0 tx=0).stop 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 -- 172.21.15.28:0/943845107 shutdown_connections 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 --2- 172.21.15.28:0/943845107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff974071a80 0x7ff97406f0a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 -- 172.21.15.28:0/943845107 >> 172.21.15.28:0/943845107 conn(0x7ff9740674f0 msgr2=0x7ff97406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 -- 172.21.15.28:0/943845107 shutdown_connections 2024-03-21T08:44:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.609+0000 7ff97b400640 1 -- 172.21.15.28:0/943845107 wait complete. 2024-03-21T08:44:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 Processor -- start 2024-03-21T08:44:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 -- start start 2024-03-21T08:44:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff974071a80 0x7ff974138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff97415cbb0 0x7ff974165e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97416a0b0 con 0x7ff9740cde40 2024-03-21T08:44:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff974169f30 con 0x7ff97415cbb0 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff97b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97416a230 con 0x7ff974071a80 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.613+0000 7ff96be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52606/0 (socket says 172.21.15.28:52606) 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 -- 172.21.15.28:0/3442591417 learned_addr learned my addr 172.21.15.28:0/3442591417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff978a00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff974071a80 0x7ff974138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff979400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff97415cbb0 0x7ff974165e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 -- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff974071a80 msgr2=0x7ff974138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff974071a80 0x7ff974138740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 -- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff97415cbb0 msgr2=0x7ff974165e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff97415cbb0 0x7ff974165e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 -- 172.21.15.28:0/3442591417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9740b2d20 con 0x7ff9740cde40 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff96be00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00c420 tx=0x7ff95c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0074e0 con 0x7ff9740cde40 2024-03-21T08:44:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c007680 con 0x7ff9740cde40 2024-03-21T08:44:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9740b3000 con 0x7ff9740cde40 2024-03-21T08:44:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c004a30 con 0x7ff9740cde40 2024-03-21T08:44:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9741584a0 con 0x7ff9740cde40 2024-03-21T08:44:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.617+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9740b31d0 con 0x7ff9740cde40 2024-03-21T08:44:02.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.621+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7ff95c01a020 con 0x7ff9740cde40 2024-03-21T08:44:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.621+0000 7ff971600640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 0x7ff950069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.621+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff95c0857f0 con 0x7ff9740cde40 2024-03-21T08:44:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.621+0000 7ff978a00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 0x7ff950069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.621+0000 7ff978a00640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 0x7ff950069fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff9640ae000 tx=0x7ff9640a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.625+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff95c056100 con 0x7ff9740cde40 2024-03-21T08:44:02.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.989+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff974073430 con 0x7ff9740cde40 2024-03-21T08:44:02.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff971600640 1 -- 172.21.15.28:0/3442591417 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff95c055a00 con 0x7ff9740cde40 2024-03-21T08:44:02.995 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 msgr2=0x7ff950069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 0x7ff950069fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff9640ae000 tx=0x7ff9640a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 msgr2=0x7ff9740b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00c420 tx=0x7ff95c00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 shutdown_connections 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7ff97415cbb0 0x7ff974165e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9740cde40 0x7ff9740b3540 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7ff950067b20 0x7ff950069fe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 --2- 172.21.15.28:0/3442591417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7ff974071a80 0x7ff974138740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 >> 172.21.15.28:0/3442591417 conn(0x7ff9740674f0 msgr2=0x7ff9740af740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 shutdown_connections 2024-03-21T08:44:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:02.993+0000 7ff97b400640 1 -- 172.21.15.28:0/3442591417 wait complete. 2024-03-21T08:44:03.026 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:44:03.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:03.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[40723]: fuse finished with error 0 and tester_r 0 2024-03-21T08:44:03.047 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:03.100 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:03.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.101 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:44:03.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-21T08:44:03.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 Processor -- start 2024-03-21T08:44:03.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 -- start start 2024-03-21T08:44:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 0x7f0550067100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055006dfc0 con 0x7f055006c890 2024-03-21T08:44:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055006fb00 con 0x7f055015bdd0 2024-03-21T08:44:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0556200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05500af810 con 0x7f055015b9f0 2024-03-21T08:44:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0555800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f055006c890 0x7f055006ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43556/0 (socket says 172.21.15.28:43556) 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0555800640 1 -- 172.21.15.28:0/2839007172 learned_addr learned my addr 172.21.15.28:0/2839007172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0554e00640 1 --2- 172.21.15.28:0/2839007172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 0x7f0550067100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855548423 0 0) 0x7f055006dfc0 con 0x7f055006c890 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0534003530 con 0x7f055006c890 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113665837 0 0) 0x7f055006fb00 con 0x7f055015bdd0 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 --> v1:172.21.15.151:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f055006dfc0 con 0x7f055015bdd0 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0554e00640 1 -- 172.21.15.28:0/2839007172 >> v1:172.21.15.151:6789/0 conn(0x7f055015bdd0 legacy=0x7f055006c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0554e00640 1 -- 172.21.15.28:0/2839007172 >> v1:172.21.15.28:6789/0 conn(0x7f055006c890 legacy=0x7f055006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.297+0000 7f0554e00640 1 -- 172.21.15.28:0/2839007172 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055006e230 con 0x7f055015b9f0 2024-03-21T08:44:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0554e00640 1 --2- 172.21.15.28:0/2839007172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 0x7f0550067100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f05400097f0 tx=0x7f054001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f22a4f83a8aab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400b0070 con 0x7f055015b9f0 2024-03-21T08:44:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400ab600 con 0x7f055015b9f0 2024-03-21T08:44:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f054fe00640 1 -- 172.21.15.28:0/2839007172 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400ab9d0 con 0x7f055015b9f0 2024-03-21T08:44:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 -- 172.21.15.28:0/2839007172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 msgr2=0x7f0550067100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 --2- 172.21.15.28:0/2839007172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 0x7f0550067100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f05400097f0 tx=0x7f054001c4e0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 -- 172.21.15.28:0/2839007172 shutdown_connections 2024-03-21T08:44:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 --2- 172.21.15.28:0/2839007172 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015b9f0 0x7f0550067100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 -- 172.21.15.28:0/2839007172 >> 172.21.15.28:0/2839007172 conn(0x7f0550072080 msgr2=0x7f0550072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 -- 172.21.15.28:0/2839007172 shutdown_connections 2024-03-21T08:44:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.301+0000 7f0556200640 1 -- 172.21.15.28:0/2839007172 wait complete. 2024-03-21T08:44:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 Processor -- start 2024-03-21T08:44:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- start start 2024-03-21T08:44:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f055006c890 0x7f0550099a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015bdd0 0x7f0550152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05500677c0 con 0x7f055015b9f0 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550067640 con 0x7f055006c890 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055016a070 con 0x7f055015bdd0 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52612/0 (socket says 172.21.15.28:52612) 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 -- 172.21.15.28:0/1291521523 learned_addr learned my addr 172.21.15.28:0/1291521523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0554e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f055006c890 0x7f0550099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0555800640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015bdd0 0x7f0550152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 -- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015bdd0 msgr2=0x7f0550152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015bdd0 0x7f0550152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 -- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f055006c890 msgr2=0x7f0550099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f055006c890 0x7f0550099a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 -- 172.21.15.28:0/1291521523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0550153b70 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0547e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f053800c430 tx=0x7f053800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538017070 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05380052f0 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538003840 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05501573a0 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.305+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0550164bc0 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.309+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0550075270 con 0x7f055015b9f0 2024-03-21T08:44:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.309+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f053801a070 con 0x7f055015b9f0 2024-03-21T08:44:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.309+0000 7f054d600640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 0x7f052c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.309+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0538085420 con 0x7f055015b9f0 2024-03-21T08:44:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.309+0000 7f0554e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 0x7f052c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.313+0000 7f0554e00640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 0x7f052c069fe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f05400ad000 tx=0x7f05400a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:03.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.313+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0550153b70 con 0x7f055015b9f0 2024-03-21T08:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.513+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0550072ad0 con 0x7f055015b9f0 2024-03-21T08:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.513+0000 7f054d600640 1 -- 172.21.15.28:0/1291521523 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7f05380555b0 con 0x7f055015b9f0 2024-03-21T08:44:03.518 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-21T08:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 msgr2=0x7f052c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 0x7f052c069fe0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f05400ad000 tx=0x7f05400a6000 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 msgr2=0x7f05501553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f053800c430 tx=0x7f053800c8f0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 shutdown_connections 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f052c067b20 0x7f052c069fe0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f055015bdd0 0x7f0550152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055015b9f0 0x7f05501553c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 --2- 172.21.15.28:0/1291521523 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f055006c890 0x7f0550099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 >> 172.21.15.28:0/1291521523 conn(0x7f0550072080 msgr2=0x7f05500b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 shutdown_connections 2024-03-21T08:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:03.517+0000 7f0556200640 1 -- 172.21.15.28:0/1291521523 wait complete. 2024-03-21T08:44:03.539 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-21T08:44:03.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-21T08:44:03.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-21T08:44:03.539 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-21T08:44:03.540 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:03.540 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-21T08:44:03.540 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-21T08:44:03.540 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-21T08:44:03.540 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:44:03.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.540 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-21T08:44:03.550 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-21T08:44:03.552 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42226sec preferred_lft 42226sec 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:28:04:9e:38 brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-21T08:44:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-21T08:44:03.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.556 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-21T08:44:03.612 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-21T08:44:03.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.615 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-21T08:44:03.668 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-21T08:44:03.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.669 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-03-21T08:44:03.669 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-21T08:44:03.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-21T08:44:03.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-21T08:44:03.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-21T08:44:03.670 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-21T08:44:03.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.761 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:44:03.770 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:44:03.770 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:44:03.771 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:44:03.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:03.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-21T08:44:03.835 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-21T08:44:03.835 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:44:03.835 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:03.887 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-21T08:44:03.887 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-21T08:44:03.887 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:03.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:44:04.003 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:44:04.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:04.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:04.060 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:44:04.111 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-21T08:44:04.111 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-21T08:44:04.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:44:04.185 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:44:04.236 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:44:04.236 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:44:04.237 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:44:04.238 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:04.239 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:04.239 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:04.239 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:04.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:04.240 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:44:04.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2024-03-21T08:44:04.257+0000 7fb62d1b4580 -1 init, newargv = 0x563c5eda8870 newargc=15 2024-03-21T08:44:04.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[41120]: starting ceph client 2024-03-21T08:44:04.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[41120]: starting fuse 2024-03-21T08:44:05.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-21T08:44:05.392 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-21T08:44:05.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-21T08:44:05.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-03-21T08:44:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-21T08:44:05.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:05.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:05.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:05.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:05.447 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:05.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:05.448 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-21T08:44:05.503 INFO:teuthology.orchestra.run.smithi028.stdout:43 2024-03-21T08:44:05.503 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:44:05.503 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-21T08:44:05.504 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-21T08:44:05.505 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:44:05.505 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2024-03-21T08:44:05.505 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:44:05.629 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.41120.asok 2024-03-21T08:44:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-21T08:44:05.641 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41120.asok 2024-03-21T08:44:05.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:05.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41120.asok status 2024-03-21T08:44:05.814 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "2edfee68ad7243bed422a049b6f5f58619300f11", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2231-g2edfee68 (2edfee68ad7243bed422a049b6f5f58619300f11) squid (dev)", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "41120", 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6588, 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6588 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1760507522 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1760507522 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.6588 192.168.0.1:0/1760507522", 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1760507522", 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 28, 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 54, 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:05.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:05.830 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-21T08:44:05.830 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:05.839 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-21T08:44:05.839 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:05.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:05.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> stdin-killer -- python3 -c ' 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> import errno 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> if True: 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> ret = os.read(fd, 1024) 2024-03-21T08:44:05.901 DEBUG:teuthology.orchestra.run.smithi028:> else: 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> os.write(fd, b'"'"'content'"'"') 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> if e.errno != errno.EPERM: 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> raise 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> else: 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("client does not check permission of data pool") 2024-03-21T08:44:05.902 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-21T08:44:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:Traceback (most recent call last): 2024-03-21T08:44:06.010 INFO:teuthology.orchestra.run.smithi028.stderr: File "", line 15, in 2024-03-21T08:44:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:RuntimeError: client does not check permission of data pool 2024-03-21T08:44:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-21T08:44:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:06.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-21T08:44:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.209+0000 7fda09000640 1 Processor -- start 2024-03-21T08:44:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.209+0000 7fda09000640 1 -- start start 2024-03-21T08:44:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.209+0000 7fda09000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda09000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda0406cdb0 0x7fda040cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda09000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040b3010 con 0x7fda040d0190 2024-03-21T08:44:06.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda09000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda040d05f0 con 0x7fda0406cdb0 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda09000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0406d310 con 0x7fda0415e1e0 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda0406cdb0 0x7fda040cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda0406cdb0 0x7fda040cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52622/0 (socket says 172.21.15.28:52622) 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fd9fbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fda040d0190 0x7fda0406c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43572/0 (socket says 172.21.15.28:43572) 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 -- 172.21.15.28:0/3755289056 learned_addr learned my addr 172.21.15.28:0/3755289056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:06.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda01e00640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 -- 172.21.15.28:0/3755289056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 msgr2=0x7fda040cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 -- 172.21.15.28:0/3755289056 >> v1:172.21.15.28:6789/0 conn(0x7fda040d0190 legacy=0x7fda0406c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 -- 172.21.15.28:0/3755289056 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0406d5f0 con 0x7fda0406cdb0 2024-03-21T08:44:06.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda01400640 1 -- 172.21.15.28:0/3755289056 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817637282 0 0) 0x7fda040b3010 con 0x7fda040d0190 2024-03-21T08:44:06.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda02800640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0406cdb0 0x7fda040cc4e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e50 tx=0x7fd9fc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9022905859dd6279 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda01400640 1 -- 172.21.15.28:0/3755289056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0a57f0 con 0x7fda0406cdb0 2024-03-21T08:44:06.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda01400640 1 -- 172.21.15.28:0/3755289056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc0a5990 con 0x7fda0406cdb0 2024-03-21T08:44:06.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.213+0000 7fda01400640 1 -- 172.21.15.28:0/3755289056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0a5d60 con 0x7fda0406cdb0 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- 172.21.15.28:0/3755289056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0406cdb0 msgr2=0x7fda040cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0406cdb0 0x7fda040cc4e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e50 tx=0x7fd9fc01ca30 comp rx=0 tx=0).stop 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- 172.21.15.28:0/3755289056 shutdown_connections 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0406cdb0 0x7fda040cc4e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- 172.21.15.28:0/3755289056 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- 172.21.15.28:0/3755289056 >> 172.21.15.28:0/3755289056 conn(0x7fda040d1ab0 msgr2=0x7fda040d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- 172.21.15.28:0/3755289056 shutdown_connections 2024-03-21T08:44:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- 172.21.15.28:0/3755289056 wait complete. 2024-03-21T08:44:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 Processor -- start 2024-03-21T08:44:06.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- start start 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 0x7fda040b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040cd5c0 con 0x7fda040d0190 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040cd440 con 0x7fda0406cdb0 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda09000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda040cd740 con 0x7fda0415e1e0 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52636/0 (socket says 172.21.15.28:52636) 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 -- 172.21.15.28:0/1632693905 learned_addr learned my addr 172.21.15.28:0/1632693905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda01e00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 0x7fda040b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda02800640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 -- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 msgr2=0x7fda040aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 -- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 msgr2=0x7fda040b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 0x7fda040b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fd9fbe00640 1 -- 172.21.15.28:0/1632693905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040af2c0 con 0x7fda040d0190 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.217+0000 7fda01e00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 0x7fda040b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fd9fbe00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00c480 tx=0x7fd9ec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec018070 con 0x7fda040d0190 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec007500 con 0x7fda040d0190 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec004a30 con 0x7fda040d0190 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda040af5a0 con 0x7fda040d0190 2024-03-21T08:44:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda040afb60 con 0x7fda040d0190 2024-03-21T08:44:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0404f4f0 con 0x7fda040d0190 2024-03-21T08:44:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.221+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd9ec004bd0 con 0x7fda040d0190 2024-03-21T08:44:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.225+0000 7fd9fa000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 0x7fd9e0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.225+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9ec0853e0 con 0x7fda040d0190 2024-03-21T08:44:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.225+0000 7fda01e00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 0x7fd9e0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.225+0000 7fda01e00640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 0x7fd9e0069fe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007a70 tx=0x7fd9f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.225+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd9ec0554f0 con 0x7fda040d0190 2024-03-21T08:44:06.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.449+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fda04073df0 con 0x7fda040d0190 2024-03-21T08:44:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.449+0000 7fd9fa000640 1 -- 172.21.15.28:0/1632693905 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) v1 ==== 184+0+141 (secure 0 0 0) 0x7fd9ec059590 con 0x7fda040d0190 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBq8vtlVs7iGBAA6asFEeRtU/M/VXVF6u4Bgw== 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-21T08:44:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 msgr2=0x7fd9e0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 0x7fd9e0069fe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007a70 tx=0x7fd9f4008040 comp rx=0 tx=0).stop 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 msgr2=0x7fda04154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00c480 tx=0x7fd9ec00c940 comp rx=0 tx=0).stop 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 shutdown_connections 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd9e0067b20 0x7fd9e0069fe0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fda0415e1e0 0x7fda040aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda040d0190 0x7fda04154dd0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 --2- 172.21.15.28:0/1632693905 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fda0406cdb0 0x7fda040b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 >> 172.21.15.28:0/1632693905 conn(0x7fda040d1ab0 msgr2=0x7fda04070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 shutdown_connections 2024-03-21T08:44:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.453+0000 7fda09000640 1 -- 172.21.15.28:0/1632693905 wait complete. 2024-03-21T08:44:06.475 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-21T08:44:06.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:44:06.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-21T08:44:06.596 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-21T08:44:06.596 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:44:06.596 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:06.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.785+0000 7f90c5200640 1 Processor -- start 2024-03-21T08:44:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.785+0000 7f90c5200640 1 -- start start 2024-03-21T08:44:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90c5200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f90c0099b80 0x7f90c009a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90c5200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 0x7f90c00d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90c5200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c00cc1d0 con 0x7f90c015e0c0 2024-03-21T08:44:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90c5200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c00d1730 con 0x7f90c0099b80 2024-03-21T08:44:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90c5200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00d1450 con 0x7f90c00cfcb0 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90bde00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90c015e0c0 0x7f90c006cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43586/0 (socket says 172.21.15.28:43586) 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90bde00640 1 -- 172.21.15.28:0/407139511 learned_addr learned my addr 172.21.15.28:0/407139511 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90b7e00640 1 --2- 172.21.15.28:0/407139511 >> v2:172.21.15.151:3300/0 conn(0x7f90c0099b80 0x7f90c009a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90be800640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 0x7f90c00d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90bd400640 1 -- 172.21.15.28:0/407139511 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834743556 0 0) 0x7f90c00cc1d0 con 0x7f90c015e0c0 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90bd400640 1 -- 172.21.15.28:0/407139511 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a4003530 con 0x7f90c015e0c0 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90b7e00640 1 -- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 msgr2=0x7f90c00d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90b7e00640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 0x7f90c00d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90b7e00640 1 -- 172.21.15.28:0/407139511 >> v1:172.21.15.28:6789/0 conn(0x7f90c015e0c0 legacy=0x7f90c006cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90b7e00640 1 -- 172.21.15.28:0/407139511 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00cc470 con 0x7f90c0099b80 2024-03-21T08:44:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.789+0000 7f90be800640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 0x7f90c00d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90b7e00640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c009a010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f90a80097c0 tx=0x7f90a801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37cdd25e7457659 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90bd400640 1 -- 172.21.15.28:0/407139511 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80b0070 con 0x7f90c0099b80 2024-03-21T08:44:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90bd400640 1 -- 172.21.15.28:0/407139511 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a80ab600 con 0x7f90c0099b80 2024-03-21T08:44:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90bd400640 1 -- 172.21.15.28:0/407139511 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80ab9d0 con 0x7f90c0099b80 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 msgr2=0x7f90c009a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c009a010 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f90a80097c0 tx=0x7f90a801c4a0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- 172.21.15.28:0/407139511 shutdown_connections 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c00cfcb0 0x7f90c00d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 --2- 172.21.15.28:0/407139511 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c009a010 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- 172.21.15.28:0/407139511 >> 172.21.15.28:0/407139511 conn(0x7f90c00b34d0 msgr2=0x7f90c00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- 172.21.15.28:0/407139511 shutdown_connections 2024-03-21T08:44:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- 172.21.15.28:0/407139511 wait complete. 2024-03-21T08:44:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 Processor -- start 2024-03-21T08:44:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.793+0000 7f90c5200640 1 -- start start 2024-03-21T08:44:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c00c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c015e0c0 0x7f90c0155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00d1010 con 0x7f90c00cfcb0 2024-03-21T08:44:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00d0e90 con 0x7f90c0099b80 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c00d1190 con 0x7f90c015e0c0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52646/0 (socket says 172.21.15.28:52646) 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 -- 172.21.15.28:0/687641580 learned_addr learned my addr 172.21.15.28:0/687641580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90be800640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c015e0c0 0x7f90c0155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90bde00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c00c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 -- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c015e0c0 msgr2=0x7f90c0155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c015e0c0 0x7f90c0155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 -- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 msgr2=0x7f90c00c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c00c5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 -- 172.21.15.28:0/687641580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00c6480 con 0x7f90c00cfcb0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b7e00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f90a801a0d0 tx=0x7f90a8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80b0040 con 0x7f90c00cfcb0 2024-03-21T08:44:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a80b9070 con 0x7f90c00cfcb0 2024-03-21T08:44:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a80b4650 con 0x7f90c00cfcb0 2024-03-21T08:44:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c01591b0 con 0x7f90c00cfcb0 2024-03-21T08:44:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c01577d0 con 0x7f90c00cfcb0 2024-03-21T08:44:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.797+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c0157310 con 0x7f90c00cfcb0 2024-03-21T08:44:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.801+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f90a80b47f0 con 0x7f90c00cfcb0 2024-03-21T08:44:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.801+0000 7f90b6000640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 0x7f909c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.801+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90a811fcf0 con 0x7f90c00cfcb0 2024-03-21T08:44:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.801+0000 7f90bde00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 0x7f909c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:06.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.801+0000 7f90bde00640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 0x7f909c069fe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005890 tx=0x7f90b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.805+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f90c0157310 con 0x7f90c00cfcb0 2024-03-21T08:44:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.969+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90c004f410 con 0x7f90c00cfcb0 2024-03-21T08:44:06.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90b6000640 1 -- 172.21.15.28:0/687641580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f90a8124070 con 0x7f90c00cfcb0 2024-03-21T08:44:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:06.975 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:06.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 msgr2=0x7f909c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 0x7f909c069fe0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005890 tx=0x7f90b0009040 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 msgr2=0x7f90c0154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f90a801a0d0 tx=0x7f90a8019240 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 shutdown_connections 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f909c067b20 0x7f909c069fe0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f90c015e0c0 0x7f90c0155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90c00cfcb0 0x7f90c0154de0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 --2- 172.21.15.28:0/687641580 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f90c0099b80 0x7f90c00c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 >> 172.21.15.28:0/687641580 conn(0x7f90c00b34d0 msgr2=0x7f90c00ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 shutdown_connections 2024-03-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:06.973+0000 7f90c5200640 1 -- 172.21.15.28:0/687641580 wait complete. 2024-03-21T08:44:06.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:44:06.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:44:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.137+0000 7fea1fc00640 1 Processor -- start 2024-03-21T08:44:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.137+0000 7fea1fc00640 1 -- start start 2024-03-21T08:44:07.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea180b0e80 0x7fea180ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea180cea00 0x7fea1808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1815fba0 con 0x7fea1815bd20 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea180cee50 con 0x7fea180cea00 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815fe80 con 0x7fea180b0e80 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea180cea00 0x7fea1808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fea1815bd20 0x7fea180b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43590/0 (socket says 172.21.15.28:43590) 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea180cea00 0x7fea1808afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52652/0 (socket says 172.21.15.28:52652) 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1d200640 1 -- 172.21.15.28:0/126611505 learned_addr learned my addr 172.21.15.28:0/126611505 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea17e00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea180b0e80 0x7fea180ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 -- 172.21.15.28:0/126611505 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea180b0e80 msgr2=0x7fea180ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea180b0e80 0x7fea180ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 -- 172.21.15.28:0/126611505 >> v1:172.21.15.28:6789/0 conn(0x7fea1815bd20 legacy=0x7fea180b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 -- 172.21.15.28:0/126611505 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18160160 con 0x7fea180cea00 2024-03-21T08:44:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea17400640 1 -- 172.21.15.28:0/126611505 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134520875 0 0) 0x7fea1815fba0 con 0x7fea1815bd20 2024-03-21T08:44:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.141+0000 7fea1dc00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180cea00 0x7fea1808afc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fea10009d90 tx=0x7fea1001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a13cbb06af5f7fc server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea17400640 1 -- 172.21.15.28:0/126611505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea100b2070 con 0x7fea180cea00 2024-03-21T08:44:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea17400640 1 -- 172.21.15.28:0/126611505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea100a5940 con 0x7fea180cea00 2024-03-21T08:44:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea17400640 1 -- 172.21.15.28:0/126611505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea100a5d10 con 0x7fea180cea00 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- 172.21.15.28:0/126611505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180cea00 msgr2=0x7fea1808afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180cea00 0x7fea1808afc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fea10009d90 tx=0x7fea1001c970 comp rx=0 tx=0).stop 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- 172.21.15.28:0/126611505 shutdown_connections 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180cea00 0x7fea1808afc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 --2- 172.21.15.28:0/126611505 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea180b0e80 0x7fea180ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- 172.21.15.28:0/126611505 >> 172.21.15.28:0/126611505 conn(0x7fea180d1ab0 msgr2=0x7fea180d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- 172.21.15.28:0/126611505 shutdown_connections 2024-03-21T08:44:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- 172.21.15.28:0/126611505 wait complete. 2024-03-21T08:44:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 Processor -- start 2024-03-21T08:44:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.145+0000 7fea1fc00640 1 -- start start 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea180cea00 0x7fea1814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea1815bd20 0x7fea1814fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1808be90 con 0x7fea180b0e80 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1808bd10 con 0x7fea180cea00 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea18168d10 con 0x7fea1815bd20 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52654/0 (socket says 172.21.15.28:52654) 2024-03-21T08:44:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 -- 172.21.15.28:0/2686709173 learned_addr learned my addr 172.21.15.28:0/2686709173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1dc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea1815bd20 0x7fea1814fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea17e00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea180cea00 0x7fea1814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 -- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea1815bd20 msgr2=0x7fea1814fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea1815bd20 0x7fea1814fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 -- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea180cea00 msgr2=0x7fea1814f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea180cea00 0x7fea1814f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 -- 172.21.15.28:0/2686709173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1806d4a0 con 0x7fea180b0e80 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1d200640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c480 tx=0x7fea0c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c018070 con 0x7fea180b0e80 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c007500 con 0x7fea180b0e80 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1806d780 con 0x7fea180b0e80 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c004a30 con 0x7fea180b0e80 2024-03-21T08:44:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1806dd40 con 0x7fea180b0e80 2024-03-21T08:44:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.149+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea18156590 con 0x7fea180b0e80 2024-03-21T08:44:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.153+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fea0c01b070 con 0x7fea180b0e80 2024-03-21T08:44:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.153+0000 7fea14c00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 0x7fe9f8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.153+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fea0c0852e0 con 0x7fea180b0e80 2024-03-21T08:44:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.153+0000 7fea17e00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 0x7fe9f8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.153+0000 7fea17e00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 0x7fe9f8069fe0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea180b7a70 tx=0x7fea04009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.157+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fea0c055500 con 0x7fea180b0e80 2024-03-21T08:44:07.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.325+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea1804f4f0 con 0x7fea180b0e80 2024-03-21T08:44:07.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.325+0000 7fea14c00640 1 -- 172.21.15.28:0/2686709173 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea0c0595a0 con 0x7fea180b0e80 2024-03-21T08:44:07.330 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:44:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 msgr2=0x7fe9f8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 0x7fe9f8069fe0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fea180b7a70 tx=0x7fea04009190 comp rx=0 tx=0).stop 2024-03-21T08:44:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 msgr2=0x7fea181f7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c480 tx=0x7fea0c00c940 comp rx=0 tx=0).stop 2024-03-21T08:44:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 shutdown_connections 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fe9f8067b20 0x7fe9f8069fe0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fea1815bd20 0x7fea1814fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fea180cea00 0x7fea1814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 --2- 172.21.15.28:0/2686709173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea180b0e80 0x7fea181f7b70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 >> 172.21.15.28:0/2686709173 conn(0x7fea180d1ab0 msgr2=0x7fea1815a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 shutdown_connections 2024-03-21T08:44:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.329+0000 7fea1fc00640 1 -- 172.21.15.28:0/2686709173 wait complete. 2024-03-21T08:44:07.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:44:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 Processor -- start 2024-03-21T08:44:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 -- start start 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 0x7fd13c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd13c0b1110 0x7fd13c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0cc3d0 con 0x7fd13c15dff0 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13c0cc180 con 0x7fd13c0b1110 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd142200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0cc6b0 con 0x7fd13c09a3a0 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd13be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd13c0b1110 0x7fd13c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd13be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd13c0b1110 0x7fd13c06d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52664/0 (socket says 172.21.15.28:52664) 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd13be00640 1 -- 172.21.15.28:0/2581237696 learned_addr learned my addr 172.21.15.28:0/2581237696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.493+0000 7fd13b400640 1 --1- 172.21.15.28:0/2581237696 >> v1:172.21.15.28:6789/0 conn(0x7fd13c15dff0 0x7fd13c099c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43606/0 (socket says 172.21.15.28:43606) 2024-03-21T08:44:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd133e00640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 0x7fd13c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13be00640 1 -- 172.21.15.28:0/2581237696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 msgr2=0x7fd13c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13be00640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 0x7fd13c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13be00640 1 -- 172.21.15.28:0/2581237696 >> v1:172.21.15.28:6789/0 conn(0x7fd13c15dff0 legacy=0x7fd13c099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13aa00640 1 -- 172.21.15.28:0/2581237696 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219112483 0 0) 0x7fd13c0cc3d0 con 0x7fd13c15dff0 2024-03-21T08:44:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13be00640 1 -- 172.21.15.28:0/2581237696 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0cc990 con 0x7fd13c0b1110 2024-03-21T08:44:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd133e00640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 0x7fd13c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13be00640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c0b1110 0x7fd13c06d140 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd134009e50 tx=0x7fd13401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6418bb76971582ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13aa00640 1 -- 172.21.15.28:0/2581237696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1340a55e0 con 0x7fd13c0b1110 2024-03-21T08:44:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13aa00640 1 -- 172.21.15.28:0/2581237696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1340a5780 con 0x7fd13c0b1110 2024-03-21T08:44:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.497+0000 7fd13aa00640 1 -- 172.21.15.28:0/2581237696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1340a5b50 con 0x7fd13c0b1110 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- 172.21.15.28:0/2581237696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c0b1110 msgr2=0x7fd13c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c0b1110 0x7fd13c06d140 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd134009e50 tx=0x7fd13401c910 comp rx=0 tx=0).stop 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- 172.21.15.28:0/2581237696 shutdown_connections 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c0b1110 0x7fd13c06d140 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 --2- 172.21.15.28:0/2581237696 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c09a3a0 0x7fd13c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- 172.21.15.28:0/2581237696 >> 172.21.15.28:0/2581237696 conn(0x7fd13c0b3620 msgr2=0x7fd13c0b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- 172.21.15.28:0/2581237696 shutdown_connections 2024-03-21T08:44:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- 172.21.15.28:0/2581237696 wait complete. 2024-03-21T08:44:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 Processor -- start 2024-03-21T08:44:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.501+0000 7fd142200640 1 -- start start 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 0x7fd13c15c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c0b1110 0x7fd13c15cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0d0a20 con 0x7fd13c15dff0 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0d08a0 con 0x7fd13c09a3a0 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0d0ba0 con 0x7fd13c0b1110 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52670/0 (socket says 172.21.15.28:52670) 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 -- 172.21.15.28:0/1366630015 learned_addr learned my addr 172.21.15.28:0/1366630015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13b400640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 0x7fd13c15c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd133e00640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c0b1110 0x7fd13c15cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 -- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c0b1110 msgr2=0x7fd13c15cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c0b1110 0x7fd13c15cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 -- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 msgr2=0x7fd13c15c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 0x7fd13c15c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 -- 172.21.15.28:0/1366630015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0c5890 con 0x7fd13c15dff0 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13b400640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 0x7fd13c15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd13be00640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd1340af040 tx=0x7fd1340196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1340bc020 con 0x7fd13c15dff0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1340b8670 con 0x7fd13c15dff0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134009510 con 0x7fd13c15dff0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c0c5b70 con 0x7fd13c15dff0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c158730 con 0x7fd13c15dff0 2024-03-21T08:44:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.505+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c0c5620 con 0x7fd13c15dff0 2024-03-21T08:44:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.509+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd1340bf020 con 0x7fd13c15dff0 2024-03-21T08:44:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.509+0000 7fd133400640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 0x7fd118069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.509+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd134120ef0 con 0x7fd13c15dff0 2024-03-21T08:44:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.509+0000 7fd13b400640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 0x7fd118069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.509+0000 7fd13b400640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 0x7fd118069fe0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd12c000f70 tx=0x7fd12c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.517+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd1340c6030 con 0x7fd13c15dff0 2024-03-21T08:44:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd13c157b50 con 0x7fd13c15dff0 2024-03-21T08:44:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd133400640 1 -- 172.21.15.28:0/1366630015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd134016620 con 0x7fd13c15dff0 2024-03-21T08:44:07.684 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 msgr2=0x7fd118069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 0x7fd118069fe0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd12c000f70 tx=0x7fd12c009040 comp rx=0 tx=0).stop 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 msgr2=0x7fd13c1f7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd1340af040 tx=0x7fd1340196b0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 shutdown_connections 2024-03-21T08:44:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd13c15dff0 0x7fd13c1f7620 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd118067b20 0x7fd118069fe0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd13c0b1110 0x7fd13c15cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 --2- 172.21.15.28:0/1366630015 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd13c09a3a0 0x7fd13c15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 >> 172.21.15.28:0/1366630015 conn(0x7fd13c0b3620 msgr2=0x7fd13c071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 shutdown_connections 2024-03-21T08:44:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.681+0000 7fd142200640 1 -- 172.21.15.28:0/1366630015 wait complete. 2024-03-21T08:44:07.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:44:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.845+0000 7f3159600640 1 Processor -- start 2024-03-21T08:44:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.845+0000 7f3159600640 1 -- start start 2024-03-21T08:44:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3159600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31540d0360 0x7f315406c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3159600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f315406cbe0 0x7f31540cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3159600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540b3010 con 0x7f315415e1e0 2024-03-21T08:44:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3159600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315406d310 con 0x7f31540d0360 2024-03-21T08:44:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3159600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315406d5f0 con 0x7f315406cbe0 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3152a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f315415e1e0 0x7f31540cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43622/0 (socket says 172.21.15.28:43622) 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3152a00640 1 -- 172.21.15.28:0/2402150434 learned_addr learned my addr 172.21.15.28:0/2402150434 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f314be00640 1 --2- 172.21.15.28:0/2402150434 >> v2:172.21.15.28:3300/0 conn(0x7f31540d0360 0x7f315406c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3153400640 1 --2- 172.21.15.28:0/2402150434 >> v2:172.21.15.151:3300/0 conn(0x7f315406cbe0 0x7f31540cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3152000640 1 -- 172.21.15.28:0/2402150434 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697573422 0 0) 0x7f31540b3010 con 0x7f315415e1e0 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f3152000640 1 -- 172.21.15.28:0/2402150434 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3138003530 con 0x7f315415e1e0 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f314be00640 1 -- 172.21.15.28:0/2402150434 >> v2:172.21.15.151:3300/0 conn(0x7f315406cbe0 msgr2=0x7f31540cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f314be00640 1 --2- 172.21.15.28:0/2402150434 >> v2:172.21.15.151:3300/0 conn(0x7f315406cbe0 0x7f31540cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f314be00640 1 -- 172.21.15.28:0/2402150434 >> v1:172.21.15.28:6789/0 conn(0x7f315415e1e0 legacy=0x7f31540cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.849+0000 7f314be00640 1 -- 172.21.15.28:0/2402150434 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315406d8d0 con 0x7f31540d0360 2024-03-21T08:44:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f314be00640 1 --2- 172.21.15.28:0/2402150434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31540d0360 0x7f315406c6a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f315406e380 tx=0x7f313c01cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca11de298dee02c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3152000640 1 -- 172.21.15.28:0/2402150434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0b1070 con 0x7f31540d0360 2024-03-21T08:44:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3152000640 1 -- 172.21.15.28:0/2402150434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c0ac5f0 con 0x7f31540d0360 2024-03-21T08:44:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3152000640 1 -- 172.21.15.28:0/2402150434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0ac9c0 con 0x7f31540d0360 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 -- 172.21.15.28:0/2402150434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31540d0360 msgr2=0x7f315406c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 --2- 172.21.15.28:0/2402150434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31540d0360 0x7f315406c6a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f315406e380 tx=0x7f313c01cee0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 -- 172.21.15.28:0/2402150434 shutdown_connections 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 --2- 172.21.15.28:0/2402150434 >> v2:172.21.15.151:3300/0 conn(0x7f315406cbe0 0x7f31540cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 --2- 172.21.15.28:0/2402150434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31540d0360 0x7f315406c6a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 -- 172.21.15.28:0/2402150434 >> 172.21.15.28:0/2402150434 conn(0x7f31540d1ab0 msgr2=0x7f31540d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 -- 172.21.15.28:0/2402150434 shutdown_connections 2024-03-21T08:44:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.853+0000 7f3159600640 1 -- 172.21.15.28:0/2402150434 wait complete. 2024-03-21T08:44:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 Processor -- start 2024-03-21T08:44:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- start start 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 0x7f315415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31540d0360 0x7f3154150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cd600 con 0x7f315415e1e0 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cd480 con 0x7f315406cbe0 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cd780 con 0x7f31540d0360 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52682/0 (socket says 172.21.15.28:52682) 2024-03-21T08:44:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 -- 172.21.15.28:0/808941923 learned_addr learned my addr 172.21.15.28:0/808941923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3152a00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 0x7f315415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f314be00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31540d0360 0x7f3154150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 -- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31540d0360 msgr2=0x7f3154150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31540d0360 0x7f3154150cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 -- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 msgr2=0x7f315415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 0x7f315415fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 -- 172.21.15.28:0/808941923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315415ff80 con 0x7f315415e1e0 2024-03-21T08:44:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3152a00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 0x7f315415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3153400640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f314c00efc0 tx=0x7f314c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c018020 con 0x7f315415e1e0 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c014480 con 0x7f315415e1e0 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c014870 con 0x7f315415e1e0 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3154160250 con 0x7f315415e1e0 2024-03-21T08:44:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.857+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3154160810 con 0x7f315415e1e0 2024-03-21T08:44:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.861+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315406fdd0 con 0x7f315415e1e0 2024-03-21T08:44:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.861+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f314c01b020 con 0x7f315415e1e0 2024-03-21T08:44:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.861+0000 7f314aa00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 0x7f3130069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.861+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f314c086ad0 con 0x7f315415e1e0 2024-03-21T08:44:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.861+0000 7f3152a00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 0x7f3130069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.865+0000 7f3152a00640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 0x7f3130069fe0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3144005890 tx=0x7f3144009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:07.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:07.865+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f314c022070 con 0x7f315415e1e0 2024-03-21T08:44:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.029+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f315404f4f0 con 0x7f315415e1e0 2024-03-21T08:44:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.029+0000 7f314aa00640 1 -- 172.21.15.28:0/808941923 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f314c010040 con 0x7f315415e1e0 2024-03-21T08:44:08.033 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 msgr2=0x7f3130069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 0x7f3130069fe0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3144005890 tx=0x7f3144009040 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 msgr2=0x7f3154147d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f314c00efc0 tx=0x7f314c00c490 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 shutdown_connections 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315415e1e0 0x7f3154147d30 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f3130067b20 0x7f3130069fe0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f31540d0360 0x7f3154150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 --2- 172.21.15.28:0/808941923 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f315406cbe0 0x7f315415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 >> 172.21.15.28:0/808941923 conn(0x7f31540d1ab0 msgr2=0x7f315415bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 shutdown_connections 2024-03-21T08:44:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.033+0000 7f3159600640 1 -- 172.21.15.28:0/808941923 wait complete. 2024-03-21T08:44:08.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-21T08:44:08.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:08.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.193+0000 7f99a9200640 1 Processor -- start 2024-03-21T08:44:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.193+0000 7f99a9200640 1 -- start start 2024-03-21T08:44:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a9200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a40cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99a40cc640 0x7f99a40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a40ccbd0 con 0x7f99a40d08f0 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40cd7e0 con 0x7f99a40cc640 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a9200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40cceb0 con 0x7f99a415bdc0 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f999be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99a40cc640 0x7f99a40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99a40d08f0 0x7f99a4099c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43636/0 (socket says 172.21.15.28:43636) 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a2800640 1 -- 172.21.15.28:0/1454449765 learned_addr learned my addr 172.21.15.28:0/1454449765 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f99a1e00640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a40cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f999be00640 1 -- 172.21.15.28:0/1454449765 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 msgr2=0x7f99a40cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f999be00640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a40cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f999be00640 1 -- 172.21.15.28:0/1454449765 >> v1:172.21.15.28:6789/0 conn(0x7f99a40d08f0 legacy=0x7f99a4099c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.197+0000 7f999be00640 1 -- 172.21.15.28:0/1454449765 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40cd190 con 0x7f99a40cc640 2024-03-21T08:44:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f999be00640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40cc640 0x7f99a40d03b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f998c0097c0 tx=0x7f998c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18855d9f559d040 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a1400640 1 -- 172.21.15.28:0/1454449765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b1070 con 0x7f99a40cc640 2024-03-21T08:44:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a1400640 1 -- 172.21.15.28:0/1454449765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0ac600 con 0x7f99a40cc640 2024-03-21T08:44:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a1400640 1 -- 172.21.15.28:0/1454449765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0ac9d0 con 0x7f99a40cc640 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 -- 172.21.15.28:0/1454449765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40cc640 msgr2=0x7f99a40d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40cc640 0x7f99a40d03b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f998c0097c0 tx=0x7f998c01c3e0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 -- 172.21.15.28:0/1454449765 shutdown_connections 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40cc640 0x7f99a40d03b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 --2- 172.21.15.28:0/1454449765 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a40cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 -- 172.21.15.28:0/1454449765 >> 172.21.15.28:0/1454449765 conn(0x7f99a40b3620 msgr2=0x7f99a40b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 -- 172.21.15.28:0/1454449765 shutdown_connections 2024-03-21T08:44:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.201+0000 7f99a9200640 1 -- 172.21.15.28:0/1454449765 wait complete. 2024-03-21T08:44:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 Processor -- start 2024-03-21T08:44:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- start start 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 0x7f99a4155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a406c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40b1b40 con 0x7f99a40d08f0 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a409a5b0 con 0x7f99a40cc640 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40b1cc0 con 0x7f99a415bdc0 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52692/0 (socket says 172.21.15.28:52692) 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 -- 172.21.15.28:0/4136389280 learned_addr learned my addr 172.21.15.28:0/4136389280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 -- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 msgr2=0x7f99a406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a1e00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 0x7f99a4155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a2800640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a406c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 -- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 msgr2=0x7f99a4155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 0x7f99a4155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 -- 172.21.15.28:0/4136389280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a415ccf0 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a1e00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 0x7f99a4155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a2800640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999be00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f998c0097c0 tx=0x7f998c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b1040 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0bc070 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a415e6d0 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b7690 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.205+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a415ec90 con 0x7f99a40d08f0 2024-03-21T08:44:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.209+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a409faf0 con 0x7f99a40d08f0 2024-03-21T08:44:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.209+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f998c0b7830 con 0x7f99a40d08f0 2024-03-21T08:44:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.209+0000 7f999a000640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 0x7f9980069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.209+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f998c120430 con 0x7f99a40d08f0 2024-03-21T08:44:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.209+0000 7f99a1e00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 0x7f9980069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.213+0000 7f99a1e00640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 0x7f9980069fe0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f99a40cc150 tx=0x7f9994008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.213+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f998c016090 con 0x7f99a40d08f0 2024-03-21T08:44:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.389+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99a404f4f0 con 0x7f99a40d08f0 2024-03-21T08:44:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.389+0000 7f999a000640 1 -- 172.21.15.28:0/4136389280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f998c0f0dd0 con 0x7f99a40d08f0 2024-03-21T08:44:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 msgr2=0x7f9980069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 0x7f9980069fe0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f99a40cc150 tx=0x7f9994008040 comp rx=0 tx=0).stop 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 msgr2=0x7f99a415c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f998c0097c0 tx=0x7f998c01ac20 comp rx=0 tx=0).stop 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 shutdown_connections 2024-03-21T08:44:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f9980067b20 0x7f9980069fe0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f99a415bdc0 0x7f99a406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a40d08f0 0x7f99a415c410 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 --2- 172.21.15.28:0/4136389280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f99a40cc640 0x7f99a4155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 >> 172.21.15.28:0/4136389280 conn(0x7f99a40b3620 msgr2=0x7f99a415a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 shutdown_connections 2024-03-21T08:44:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.393+0000 7f99a9200640 1 -- 172.21.15.28:0/4136389280 wait complete. 2024-03-21T08:44:08.414 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-21T08:44:08.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:08.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-21T08:44:08.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[41120]: fuse finished with error 0 and tester_r 0 2024-03-21T08:44:08.435 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:08.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:08.435 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-21T08:44:08.600 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:44:08.600 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:08.609 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:44:08.609 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:08.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:08.609 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-21T08:44:08.661 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-21T08:44:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:08.662 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:44:08.662 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:08.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:44:08.671 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:08.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:08.672 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-21T08:44:08.721 INFO:teuthology.orchestra.run.smithi028.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-21T08:44:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:08.722 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:44:08.722 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:08.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:44:08.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:08.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:08.732 DEBUG:teuthology.orchestra.run.smithi151:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-21T08:44:08.781 INFO:teuthology.orchestra.run.smithi151.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-21T08:44:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:08.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:44:08.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 Processor -- start 2024-03-21T08:44:08.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 -- start start 2024-03-21T08:44:08.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f98d80cedb0 0x7f98d80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 0x7f98d80cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d8160590 con 0x7f98d815bbe0 2024-03-21T08:44:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d815fd20 con 0x7f98d80cedb0 2024-03-21T08:44:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98e0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d815ffd0 con 0x7f98d80cf970 2024-03-21T08:44:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98dd600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98d815bbe0 0x7f98d815f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43638/0 (socket says 172.21.15.28:43638) 2024-03-21T08:44:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98dd600640 1 -- 172.21.15.28:0/2830876193 learned_addr learned my addr 172.21.15.28:0/2830876193 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98de000640 1 --2- 172.21.15.28:0/2830876193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 0x7f98d80cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98dcc00640 1 --2- 172.21.15.28:0/2830876193 >> v2:172.21.15.151:3300/0 conn(0x7f98d80cedb0 0x7f98d80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98d7e00640 1 -- 172.21.15.28:0/2830876193 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840562532 0 0) 0x7f98d8160590 con 0x7f98d815bbe0 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98d7e00640 1 -- 172.21.15.28:0/2830876193 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c0003530 con 0x7f98d815bbe0 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98de000640 1 -- 172.21.15.28:0/2830876193 >> v2:172.21.15.151:3300/0 conn(0x7f98d80cedb0 msgr2=0x7f98d80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98de000640 1 --2- 172.21.15.28:0/2830876193 >> v2:172.21.15.151:3300/0 conn(0x7f98d80cedb0 0x7f98d80cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98de000640 1 -- 172.21.15.28:0/2830876193 >> v1:172.21.15.28:6789/0 conn(0x7f98d815bbe0 legacy=0x7f98d815f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.921+0000 7f98de000640 1 -- 172.21.15.28:0/2830876193 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d81602b0 con 0x7f98d80cf970 2024-03-21T08:44:08.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98de000640 1 --2- 172.21.15.28:0/2830876193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 0x7f98d80cfe00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d90 tx=0x7f98d001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a53a92af3c00aa42 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98d7e00640 1 -- 172.21.15.28:0/2830876193 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00b2070 con 0x7f98d80cf970 2024-03-21T08:44:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98d7e00640 1 -- 172.21.15.28:0/2830876193 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d00a5970 con 0x7f98d80cf970 2024-03-21T08:44:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98d7e00640 1 -- 172.21.15.28:0/2830876193 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00a5d40 con 0x7f98d80cf970 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 -- 172.21.15.28:0/2830876193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 msgr2=0x7f98d80cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 --2- 172.21.15.28:0/2830876193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 0x7f98d80cfe00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d90 tx=0x7f98d001c8b0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 -- 172.21.15.28:0/2830876193 shutdown_connections 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 --2- 172.21.15.28:0/2830876193 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d80cf970 0x7f98d80cfe00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 --2- 172.21.15.28:0/2830876193 >> v2:172.21.15.151:3300/0 conn(0x7f98d80cedb0 0x7f98d80cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 -- 172.21.15.28:0/2830876193 >> 172.21.15.28:0/2830876193 conn(0x7f98d8099a30 msgr2=0x7f98d8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 -- 172.21.15.28:0/2830876193 shutdown_connections 2024-03-21T08:44:08.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.925+0000 7f98e0000640 1 -- 172.21.15.28:0/2830876193 wait complete. 2024-03-21T08:44:08.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 Processor -- start 2024-03-21T08:44:08.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- start start 2024-03-21T08:44:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98d80cf970 0x7f98d80b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d815bbe0 0x7f98d815d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8168b30 con 0x7f98d80cedb0 2024-03-21T08:44:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d81689b0 con 0x7f98d80cf970 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8168cb0 con 0x7f98d815bbe0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52706/0 (socket says 172.21.15.28:52706) 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 -- 172.21.15.28:0/1849351702 learned_addr learned my addr 172.21.15.28:0/1849351702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dcc00640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98d80cf970 0x7f98d80b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98de000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d815bbe0 0x7f98d815d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 -- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d815bbe0 msgr2=0x7f98d815d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d815bbe0 0x7f98d815d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 -- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98d80cf970 msgr2=0x7f98d80b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98d80cf970 0x7f98d80b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 -- 172.21.15.28:0/1849351702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8159850 con 0x7f98d80cedb0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98dd600640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00c3f0 tx=0x7f98cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc017070 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc005cf0 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0072f0 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d80b34d0 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.929+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d80b3a90 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d804f410 con 0x7f98d80cedb0 2024-03-21T08:44:08.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f98cc01a070 con 0x7f98d80cedb0 2024-03-21T08:44:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98d5600640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 0x7f98b8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f98cc085540 con 0x7f98d80cedb0 2024-03-21T08:44:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98dcc00640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 0x7f98b8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:08.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.933+0000 7f98dcc00640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 0x7f98b8069fe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f98d80ca3e0 tx=0x7f98c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:08.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:08.937+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f98cc055650 con 0x7f98d80cedb0 2024-03-21T08:44:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.137+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d80cd420 con 0x7f98d80cedb0 2024-03-21T08:44:09.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.137+0000 7f98d5600640 1 -- 172.21.15.28:0/1849351702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f98cc0596f0 con 0x7f98d80cedb0 2024-03-21T08:44:09.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:44:09.142 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 msgr2=0x7f98b8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 0x7f98b8069fe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f98d80ca3e0 tx=0x7f98c4009190 comp rx=0 tx=0).stop 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 msgr2=0x7f98d8159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00c3f0 tx=0x7f98cc00c8b0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 shutdown_connections 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f98b8067b20 0x7f98b8069fe0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f98d815bbe0 0x7f98d815d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f98d80cf970 0x7f98d80b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 --2- 172.21.15.28:0/1849351702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cedb0 0x7f98d8159310 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 >> 172.21.15.28:0/1849351702 conn(0x7f98d8099a30 msgr2=0x7f98d8165ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 shutdown_connections 2024-03-21T08:44:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.141+0000 7f98e0000640 1 -- 172.21.15.28:0/1849351702 wait complete. 2024-03-21T08:44:09.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.305+0000 7f1c8ae00640 1 Processor -- start 2024-03-21T08:44:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.305+0000 7f1c8ae00640 1 -- start start 2024-03-21T08:44:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c8ae00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1c840ce490 0x7f1c8415e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c8ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c840b4380 0x7f1c840b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c8ae00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8415bf00 con 0x7f1c8415ba70 2024-03-21T08:44:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c8ae00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8415ebf0 con 0x7f1c840ce490 2024-03-21T08:44:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c8ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8415eed0 con 0x7f1c840b4380 2024-03-21T08:44:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1c840ce490 0x7f1c8415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c88e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c840b4380 0x7f1c840b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c7be00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f1c8415ba70 0x7f1c840ce870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:56912/0 (socket says 172.21.15.28:56912) 2024-03-21T08:44:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f1c840ce490 0x7f1c8415e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:51082/0 (socket says 172.21.15.28:51082) 2024-03-21T08:44:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c7be00640 1 -- 172.21.15.28:0/1238786649 learned_addr learned my addr 172.21.15.28:0/1238786649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 -- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c840b4380 msgr2=0x7f1c840b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 --2- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c840b4380 0x7f1c840b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 -- 172.21.15.28:0/1238786649 >> v1:172.21.15.151:6789/0 conn(0x7f1c8415ba70 legacy=0x7f1c840ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83e00640 1 -- 172.21.15.28:0/1238786649 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8415f1b0 con 0x7f1c840ce490 2024-03-21T08:44:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.309+0000 7f1c83400640 1 -- 172.21.15.28:0/1238786649 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804439630 0 0) 0x7f1c8415bf00 con 0x7f1c8415ba70 2024-03-21T08:44:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c83e00640 1 --2- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840ce490 0x7f1c8415e610 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c740097c0 tx=0x7f1c7401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ea9caefffef438b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c83400640 1 -- 172.21.15.28:0/1238786649 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c740b0070 con 0x7f1c840ce490 2024-03-21T08:44:09.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c83400640 1 -- 172.21.15.28:0/1238786649 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c740ab600 con 0x7f1c840ce490 2024-03-21T08:44:09.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c83400640 1 -- 172.21.15.28:0/1238786649 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c740ab9d0 con 0x7f1c840ce490 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 -- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840ce490 msgr2=0x7f1c8415e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840ce490 0x7f1c8415e610 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c740097c0 tx=0x7f1c7401c410 comp rx=0 tx=0).stop 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 -- 172.21.15.28:0/1238786649 shutdown_connections 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c840b4380 0x7f1c840b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/1238786649 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840ce490 0x7f1c8415e610 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 -- 172.21.15.28:0/1238786649 >> 172.21.15.28:0/1238786649 conn(0x7f1c840cc230 msgr2=0x7f1c840cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 -- 172.21.15.28:0/1238786649 shutdown_connections 2024-03-21T08:44:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.313+0000 7f1c8ae00640 1 -- 172.21.15.28:0/1238786649 wait complete. 2024-03-21T08:44:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 Processor -- start 2024-03-21T08:44:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 -- start start 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840b4380 0x7f1c84154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c8415ba70 0x7f1c84159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c84167b30 con 0x7f1c840ce490 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c841679b0 con 0x7f1c840b4380 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c8ae00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c84167cb0 con 0x7f1c8415ba70 2024-03-21T08:44:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52714/0 (socket says 172.21.15.28:52714) 2024-03-21T08:44:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 -- 172.21.15.28:0/484647009 learned_addr learned my addr 172.21.15.28:0/484647009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c83e00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840b4380 0x7f1c84154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c88e00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c8415ba70 0x7f1c84159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 -- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c8415ba70 msgr2=0x7f1c84159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c8415ba70 0x7f1c84159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 -- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840b4380 msgr2=0x7f1c84154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840b4380 0x7f1c84154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 -- 172.21.15.28:0/484647009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8406dd00 con 0x7f1c840ce490 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.317+0000 7f1c7be00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007fd0 tx=0x7f1c6c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c017070 con 0x7f1c840ce490 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c005d10 con 0x7f1c840ce490 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c007310 con 0x7f1c840ce490 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8406dfe0 con 0x7f1c840ce490 2024-03-21T08:44:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8406e5a0 con 0x7f1c840ce490 2024-03-21T08:44:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8404f410 con 0x7f1c840ce490 2024-03-21T08:44:09.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.321+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f1c6c01a070 con 0x7f1c840ce490 2024-03-21T08:44:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.325+0000 7f1c80c00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 0x7f1c60069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.325+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c6c0854b0 con 0x7f1c840ce490 2024-03-21T08:44:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.325+0000 7f1c83e00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 0x7f1c60069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.325+0000 7f1c83e00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 0x7f1c60069fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c740ad000 tx=0x7f1c74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.329+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1c6c055640 con 0x7f1c840ce490 2024-03-21T08:44:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.629+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c840cbfa0 con 0x7f1c840ce490 2024-03-21T08:44:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.629+0000 7f1c80c00640 1 -- 172.21.15.28:0/484647009 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f1c6c0596e0 con 0x7f1c840ce490 2024-03-21T08:44:09.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:09.634 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 msgr2=0x7f1c60069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 0x7f1c60069fe0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c740ad000 tx=0x7f1c74008040 comp rx=0 tx=0).stop 2024-03-21T08:44:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 msgr2=0x7f1c841536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007fd0 tx=0x7f1c6c00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 shutdown_connections 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1c60067b20 0x7f1c60069fe0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1c8415ba70 0x7f1c84159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c840ce490 0x7f1c841536b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 --2- 172.21.15.28:0/484647009 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1c840b4380 0x7f1c84154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 >> 172.21.15.28:0/484647009 conn(0x7f1c840cc230 msgr2=0x7f1c84071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 shutdown_connections 2024-03-21T08:44:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.633+0000 7f1c8ae00640 1 -- 172.21.15.28:0/484647009 wait complete. 2024-03-21T08:44:09.656 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T08:40:53.789978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-21T08:43:31.890980+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-21T08:43:32.446780+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T08:44:09.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-21T08:44:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 Processor -- start 2024-03-21T08:44:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 -- start start 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b880b0e80 0x7f0b880ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b8808afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8815fba0 con 0x7f0b8815bd20 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b880cee30 con 0x7f0b880b0e80 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b8e200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8815fe80 con 0x7f0b880cea00 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b7fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b880b0e80 0x7f0b880ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b87400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0b8815bd20 0x7f0b880b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43648/0 (socket says 172.21.15.28:43648) 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b87400640 1 -- 172.21.15.28:0/296905332 learned_addr learned my addr 172.21.15.28:0/296905332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b87e00640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b8808afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b86a00640 1 -- 172.21.15.28:0/296905332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223781827 0 0) 0x7f0b8815fba0 con 0x7f0b8815bd20 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b86a00640 1 -- 172.21.15.28:0/296905332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b6c003530 con 0x7f0b8815bd20 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b7fe00640 1 -- 172.21.15.28:0/296905332 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 msgr2=0x7f0b8808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b8808afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b7fe00640 1 -- 172.21.15.28:0/296905332 >> v1:172.21.15.28:6789/0 conn(0x7f0b8815bd20 legacy=0x7f0b880b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b7fe00640 1 -- 172.21.15.28:0/296905332 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88160160 con 0x7f0b880b0e80 2024-03-21T08:44:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.797+0000 7f0b87e00640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b8808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b880ce4c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0b700097c0 tx=0x7f0b7001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f6ac88c19b71d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b86a00640 1 -- 172.21.15.28:0/296905332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b700b0070 con 0x7f0b880b0e80 2024-03-21T08:44:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b86a00640 1 -- 172.21.15.28:0/296905332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b700ab600 con 0x7f0b880b0e80 2024-03-21T08:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b86a00640 1 -- 172.21.15.28:0/296905332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b700ab9b0 con 0x7f0b880b0e80 2024-03-21T08:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 -- 172.21.15.28:0/296905332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 msgr2=0x7f0b880ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b880ce4c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0b700097c0 tx=0x7f0b7001c4a0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 -- 172.21.15.28:0/296905332 shutdown_connections 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b8808afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 --2- 172.21.15.28:0/296905332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b880ce4c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 -- 172.21.15.28:0/296905332 >> 172.21.15.28:0/296905332 conn(0x7f0b880d1ab0 msgr2=0x7f0b880d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 -- 172.21.15.28:0/296905332 shutdown_connections 2024-03-21T08:44:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.801+0000 7f0b8e200640 1 -- 172.21.15.28:0/296905332 wait complete. 2024-03-21T08:44:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 Processor -- start 2024-03-21T08:44:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- start start 2024-03-21T08:44:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b881569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b8815bd20 0x7f0b88159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8808bd10 con 0x7f0b880b0e80 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8815c310 con 0x7f0b8815bd20 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8808be90 con 0x7f0b880cea00 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52722/0 (socket says 172.21.15.28:52722) 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 -- 172.21.15.28:0/1006759242 learned_addr learned my addr 172.21.15.28:0/1006759242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87e00640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b8815bd20 0x7f0b88159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b881569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 -- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 msgr2=0x7f0b881569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b881569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 -- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b8815bd20 msgr2=0x7f0b88159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b8815bd20 0x7f0b88159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 -- 172.21.15.28:0/1006759242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88157480 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b881569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b87400640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800c460 tx=0x7f0b7800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78017070 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b78005d10 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b780072b0 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88157760 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.805+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88157d20 con 0x7f0b880b0e80 2024-03-21T08:44:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8804f4f0 con 0x7f0b880b0e80 2024-03-21T08:44:09.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0b7801a070 con 0x7f0b880b0e80 2024-03-21T08:44:09.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b7f400640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 0x7f0b6406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:09.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0b780854b0 con 0x7f0b880b0e80 2024-03-21T08:44:09.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 0x7f0b6406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.809+0000 7f0b7fe00640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 0x7f0b6406a010 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0b700ad000 tx=0x7f0b70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:09.813+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0b780555b0 con 0x7f0b880b0e80 2024-03-21T08:44:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.033+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b880d2830 con 0x7f0b880b0e80 2024-03-21T08:44:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.033+0000 7f0b7f400640 1 -- 172.21.15.28:0/1006759242 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b78059650 con 0x7f0b880b0e80 2024-03-21T08:44:10.037 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-21T08:44:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 msgr2=0x7f0b6406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 0x7f0b6406a010 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0b700ad000 tx=0x7f0b70008040 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 msgr2=0x7f0b8814ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800c460 tx=0x7f0b7800c920 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 shutdown_connections 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0b8815bd20 0x7f0b88159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0b64067b50 0x7f0b6406a010 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0b880cea00 0x7f0b881569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 --2- 172.21.15.28:0/1006759242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b880b0e80 0x7f0b8814ede0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 >> 172.21.15.28:0/1006759242 conn(0x7f0b880d1ab0 msgr2=0x7f0b8815aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 shutdown_connections 2024-03-21T08:44:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.037+0000 7f0b8e200640 1 -- 172.21.15.28:0/1006759242 wait complete. 2024-03-21T08:44:10.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-21T08:44:10.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.201+0000 7f32ce200640 1 Processor -- start 2024-03-21T08:44:10.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.201+0000 7f32ce200640 1 -- start start 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32ce200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f32c80b0e80 0x7f32c80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32ce200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80cede0 con 0x7f32c815bd20 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32ce200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c815fba0 con 0x7f32c80cea00 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32ce200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c815fe80 con 0x7f32c80b0e80 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32c815bd20 0x7f32c80b2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43654/0 (socket says 172.21.15.28:43654) 2024-03-21T08:44:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7400640 1 -- 172.21.15.28:0/2563977372 learned_addr learned my addr 172.21.15.28:0/2563977372 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6a00640 1 --2- 172.21.15.28:0/2563977372 >> v2:172.21.15.151:3300/0 conn(0x7f32c80b0e80 0x7f32c80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185703693 0 0) 0x7f32c80cede0 con 0x7f32c815bd20 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003530 con 0x7f32c815bd20 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991474505 0 0) 0x7f32ac003530 con 0x7f32c815bd20 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 >> v2:172.21.15.151:3300/0 conn(0x7f32c80b0e80 msgr2=0x7f32c80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 --2- 172.21.15.28:0/2563977372 >> v2:172.21.15.151:3300/0 conn(0x7f32c80b0e80 0x7f32c80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 >> v1:172.21.15.151:6789/0 conn(0x7f32c80cea00 legacy=0x7f32c808afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8160160 con 0x7f32c815bd20 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f32b8001680 con 0x7f32c815bd20 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 >> v1:172.21.15.28:6789/0 conn(0x7f32c815bd20 legacy=0x7f32c80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32ac0039c0 0x7f32ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac004810 0x7f32ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 0x7f32ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c815fe80 con 0x7f32ac0051d0 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00c070 con 0x7f32ac004810 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00c270 con 0x7f32ac0039c0 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 0x7f32ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6a00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac004810 0x7f32ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7400640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32ac0039c0 0x7f32ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 -- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32ac0039c0 msgr2=0x7f32ac003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32ac0039c0 0x7f32ac003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 -- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac004810 msgr2=0x7f32ac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac004810 0x7f32ac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c7e00640 1 -- 172.21.15.28:0/2563977372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac003530 con 0x7f32ac0051d0 2024-03-21T08:44:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.205+0000 7f32c6a00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac004810 0x7f32ac004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:10.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.209+0000 7f32c7e00640 1 -- 172.21.15.28:0/2563977372 reap_dead start 2024-03-21T08:44:10.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.209+0000 7f32c7e00640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 0x7f32ac00b9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f32c0001e70 tx=0x7f32c001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d154da4eee40ef0c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.209+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b40096b0 con 0x7f32ac0051d0 2024-03-21T08:44:10.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.209+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0003bc0 con 0x7f32ac0051d0 2024-03-21T08:44:10.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.209+0000 7f32c6000640 1 -- 172.21.15.28:0/2563977372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0001a90 con 0x7f32ac0051d0 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 msgr2=0x7f32ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 0x7f32ac00b9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f32c0001e70 tx=0x7f32c001c780 comp rx=0 tx=0).stop 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- 172.21.15.28:0/2563977372 shutdown_connections 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 --2- 172.21.15.28:0/2563977372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32ac0051d0 0x7f32ac00b9d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- 172.21.15.28:0/2563977372 >> 172.21.15.28:0/2563977372 conn(0x7f32c80d1ab0 msgr2=0x7f32c80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- 172.21.15.28:0/2563977372 shutdown_connections 2024-03-21T08:44:10.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- 172.21.15.28:0/2563977372 wait complete. 2024-03-21T08:44:10.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 Processor -- start 2024-03-21T08:44:10.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- start start 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac0051d0 0x7f32c815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c806d070 0x7f32c80c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80cc6a0 con 0x7f32c815ecf0 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80cc520 con 0x7f32ac0051d0 2024-03-21T08:44:10.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32ce200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c815c100 con 0x7f32c806d070 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52730/0 (socket says 172.21.15.28:52730) 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32c6a00640 1 -- 172.21.15.28:0/8629876 learned_addr learned my addr 172.21.15.28:0/8629876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32c7400640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac0051d0 0x7f32c815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.213+0000 7f32c7e00640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c806d070 0x7f32c80c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 -- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c806d070 msgr2=0x7f32c80c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c806d070 0x7f32c80c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 -- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac0051d0 msgr2=0x7f32c815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac0051d0 0x7f32c815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 -- 172.21.15.28:0/8629876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c815dc50 con 0x7f32c815ecf0 2024-03-21T08:44:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32c6a00640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f32b4011960 tx=0x7f32b4011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b400fa30 con 0x7f32c815ecf0 2024-03-21T08:44:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b400fbd0 con 0x7f32c815ecf0 2024-03-21T08:44:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4004020 con 0x7f32c815ecf0 2024-03-21T08:44:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c815df30 con 0x7f32c815ecf0 2024-03-21T08:44:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c815e4f0 con 0x7f32c815ecf0 2024-03-21T08:44:10.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c804f4f0 con 0x7f32c815ecf0 2024-03-21T08:44:10.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f32b4012040 con 0x7f32c815ecf0 2024-03-21T08:44:10.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 0x7f32a8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.217+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32b4086300 con 0x7f32c815ecf0 2024-03-21T08:44:10.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.221+0000 7f32c7400640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 0x7f32a8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.221+0000 7f32c7400640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 0x7f32a8069fe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f32c815f290 tx=0x7f32b8001c70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.221+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f32b4019030 con 0x7f32c815ecf0 2024-03-21T08:44:10.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.385+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32c80d2830 con 0x7f32c815ecf0 2024-03-21T08:44:10.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.385+0000 7f32bf400640 1 -- 172.21.15.28:0/8629876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32b4056520 con 0x7f32c815ecf0 2024-03-21T08:44:10.390 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-21T08:44:10.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 msgr2=0x7f32a8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 0x7f32a8069fe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f32c815f290 tx=0x7f32b8001c70 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 msgr2=0x7f32c806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f32b4011960 tx=0x7f32b4011e20 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 shutdown_connections 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f32a8067b20 0x7f32a8069fe0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f32c806d070 0x7f32c80c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c815ecf0 0x7f32c806dc00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 --2- 172.21.15.28:0/8629876 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f32ac0051d0 0x7f32c815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 >> 172.21.15.28:0/8629876 conn(0x7f32c80d1ab0 msgr2=0x7f32c80cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 shutdown_connections 2024-03-21T08:44:10.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.389+0000 7f32ce200640 1 -- 172.21.15.28:0/8629876 wait complete. 2024-03-21T08:44:10.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-21T08:44:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.557+0000 7f41ef200640 1 Processor -- start 2024-03-21T08:44:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.557+0000 7f41ef200640 1 -- start start 2024-03-21T08:44:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41ef200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e815bd20 0x7f41e80b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41ef200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f41e80b0e80 0x7f41e80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41ef200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e815fba0 con 0x7f41e80cea00 2024-03-21T08:44:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41ef200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e808bfd0 con 0x7f41e80b0e80 2024-03-21T08:44:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41ef200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80cee50 con 0x7f41e815bd20 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f41e80b0e80 0x7f41e80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f41e80b0e80 0x7f41e80ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:56996/0 (socket says 172.21.15.28:56996) 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 -- 172.21.15.28:0/3253300691 learned_addr learned my addr 172.21.15.28:0/3253300691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41e7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e815bd20 0x7f41e80b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 -- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e815bd20 msgr2=0x7f41e80b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 --2- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e815bd20 0x7f41e80b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 -- 172.21.15.28:0/3253300691 >> v1:172.21.15.151:6789/0 conn(0x7f41e80cea00 legacy=0x7f41e80b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.561+0000 7f41df400640 1 -- 172.21.15.28:0/3253300691 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e815fe80 con 0x7f41e80b0e80 2024-03-21T08:44:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41df400640 1 --2- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80b0e80 0x7f41e80ce4c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009870 tx=0x7f41d001c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21c11e7b94a5d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41e7400640 1 -- 172.21.15.28:0/3253300691 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00b0070 con 0x7f41e80b0e80 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41e7400640 1 -- 172.21.15.28:0/3253300691 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d00ab600 con 0x7f41e80b0e80 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41e7400640 1 -- 172.21.15.28:0/3253300691 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00ab9d0 con 0x7f41e80b0e80 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 -- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80b0e80 msgr2=0x7f41e80ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 --2- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80b0e80 0x7f41e80ce4c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009870 tx=0x7f41d001c560 comp rx=0 tx=0).stop 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 -- 172.21.15.28:0/3253300691 shutdown_connections 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 --2- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80b0e80 0x7f41e80ce4c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 --2- 172.21.15.28:0/3253300691 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e815bd20 0x7f41e80b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 -- 172.21.15.28:0/3253300691 >> 172.21.15.28:0/3253300691 conn(0x7f41e80d1ab0 msgr2=0x7f41e80d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 -- 172.21.15.28:0/3253300691 shutdown_connections 2024-03-21T08:44:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.565+0000 7f41ef200640 1 -- 172.21.15.28:0/3253300691 wait complete. 2024-03-21T08:44:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 Processor -- start 2024-03-21T08:44:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- start start 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 0x7f41e8153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 0x7f41e80cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80cc6a0 con 0x7f41e815bd20 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80cc520 con 0x7f41e80cea00 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e815c100 con 0x7f41e80b0e80 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ed200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 0x7f41e80cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ed200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 0x7f41e80cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43730/0 (socket says 172.21.15.28:43730) 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:42070/0 (socket says 172.21.15.28:42070) 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41df400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 0x7f41e8153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ed200640 1 -- 172.21.15.28:0/1188768081 learned_addr learned my addr 172.21.15.28:0/1188768081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 -- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 msgr2=0x7f41e8153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 0x7f41e8153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 -- 172.21.15.28:0/1188768081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 msgr2=0x7f41e80cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 0x7f41e80cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 -- 172.21.15.28:0/1188768081 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e815a360 con 0x7f41e80b0e80 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41df400640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 0x7f41e8153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e7e00640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e870 tx=0x7f41d800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8017070 con 0x7f41e80b0e80 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d800c940 con 0x7f41e80b0e80 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d800ccf0 con 0x7f41e80b0e80 2024-03-21T08:44:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e815a640 con 0x7f41e80b0e80 2024-03-21T08:44:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e815ac00 con 0x7f41e80b0e80 2024-03-21T08:44:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.569+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e804f4f0 con 0x7f41e80b0e80 2024-03-21T08:44:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.573+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f41d801a070 con 0x7f41e80b0e80 2024-03-21T08:44:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.573+0000 7f41e4c00640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 0x7f41c406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.573+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f41d8086780 con 0x7f41e80b0e80 2024-03-21T08:44:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.573+0000 7f41df400640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 0x7f41c406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.573+0000 7f41df400640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 0x7f41c406a010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f41d00ad000 tx=0x7f41d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.577+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f41d8056990 con 0x7f41e80b0e80 2024-03-21T08:44:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.741+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41e80d2830 con 0x7f41e80b0e80 2024-03-21T08:44:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.741+0000 7f41e4c00640 1 -- 172.21.15.28:0/1188768081 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41d805aa30 con 0x7f41e80b0e80 2024-03-21T08:44:10.744 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 msgr2=0x7f41c406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 0x7f41c406a010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f41d00ad000 tx=0x7f41d0008040 comp rx=0 tx=0).stop 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 msgr2=0x7f41e81520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e870 tx=0x7f41d800ed30 comp rx=0 tx=0).stop 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 shutdown_connections 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e815bd20 0x7f41e80cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f41e80cea00 0x7f41e8153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f41c4067b50 0x7f41c406a010 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 --2- 172.21.15.28:0/1188768081 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f41e80b0e80 0x7f41e81520c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 >> 172.21.15.28:0/1188768081 conn(0x7f41e80d1ab0 msgr2=0x7f41e8073b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 shutdown_connections 2024-03-21T08:44:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.745+0000 7f41ef200640 1 -- 172.21.15.28:0/1188768081 wait complete. 2024-03-21T08:44:10.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:44:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 Processor -- start 2024-03-21T08:44:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 -- start start 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a78099fe0 0x7f6a780c6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780cdb50 con 0x7f6a78099b80 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780cde30 con 0x7f6a7815f260 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a780c5530 con 0x7f6a78099fe0 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a6f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a78099b80 0x7f6a780c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39926/0 (socket says 172.21.15.28:39926) 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a6f400640 1 -- 172.21.15.28:0/2050522683 learned_addr learned my addr 172.21.15.28:0/2050522683 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ce00640 1 --2- 172.21.15.28:0/2050522683 >> v2:172.21.15.28:3300/0 conn(0x7f6a78099fe0 0x7f6a780c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a77400640 1 -- 172.21.15.28:0/2050522683 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613793276 0 0) 0x7f6a780cdb50 con 0x7f6a78099b80 2024-03-21T08:44:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a77400640 1 -- 172.21.15.28:0/2050522683 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a5c003530 con 0x7f6a78099b80 2024-03-21T08:44:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ce00640 1 -- 172.21.15.28:0/2050522683 >> v1:172.21.15.151:6789/0 conn(0x7f6a7815f260 legacy=0x7f6a780cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ce00640 1 -- 172.21.15.28:0/2050522683 >> v1:172.21.15.28:6789/0 conn(0x7f6a78099b80 legacy=0x7f6a780c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.905+0000 7f6a7ce00640 1 -- 172.21.15.28:0/2050522683 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780c5780 con 0x7f6a78099fe0 2024-03-21T08:44:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ce00640 1 --2- 172.21.15.28:0/2050522683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099fe0 0x7f6a780c6690 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009d90 tx=0x7f6a7001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731aa1bce67835f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a77400640 1 -- 172.21.15.28:0/2050522683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700b2070 con 0x7f6a78099fe0 2024-03-21T08:44:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a77400640 1 -- 172.21.15.28:0/2050522683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a700a5970 con 0x7f6a78099fe0 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a77400640 1 -- 172.21.15.28:0/2050522683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700a5d20 con 0x7f6a78099fe0 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2050522683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099fe0 msgr2=0x7f6a780c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2050522683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099fe0 0x7f6a780c6690 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009d90 tx=0x7f6a7001c8b0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2050522683 shutdown_connections 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2050522683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099fe0 0x7f6a780c6690 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2050522683 >> 172.21.15.28:0/2050522683 conn(0x7f6a780b34d0 msgr2=0x7f6a780b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2050522683 shutdown_connections 2024-03-21T08:44:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2050522683 wait complete. 2024-03-21T08:44:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 Processor -- start 2024-03-21T08:44:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 -- start start 2024-03-21T08:44:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a78099fe0 0x7f6a78157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.913+0000 7f6a7ee00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6a7815f260 0x7f6a7814e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780cd7a0 con 0x7f6a78099b80 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7815f640 con 0x7f6a78099fe0 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7806d470 con 0x7f6a7815f260 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43744/0 (socket says 172.21.15.28:43744) 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 -- 172.21.15.28:0/2148312067 learned_addr learned my addr 172.21.15.28:0/2148312067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a6f400640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a78099fe0 0x7f6a78157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ce00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6a7815f260 0x7f6a7814e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 -- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6a7815f260 msgr2=0x7f6a7814e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6a7815f260 0x7f6a7814e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 -- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a78099fe0 msgr2=0x7f6a78157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a78099fe0 0x7f6a78157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 -- 172.21.15.28:0/2148312067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7806c5c0 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a77e00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c3c0 tx=0x7f6a6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68017070 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a680052d0 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7806c860 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68003840 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7806d100 con 0x7f6a78099b80 2024-03-21T08:44:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7804f410 con 0x7f6a78099b80 2024-03-21T08:44:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6a6801a070 con 0x7f6a78099b80 2024-03-21T08:44:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.917+0000 7f6a74c00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 0x7f6a54069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.921+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a680853a0 con 0x7f6a78099b80 2024-03-21T08:44:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.921+0000 7f6a6f400640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 0x7f6a54069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.921+0000 7f6a6f400640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 0x7f6a54069fe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a7814fa70 tx=0x7f6a60007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:10.925+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6a68005480 con 0x7f6a78099b80 2024-03-21T08:44:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a7806de40 con 0x7f6a78099b80 2024-03-21T08:44:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a74c00640 1 -- 172.21.15.28:0/2148312067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f6a68055530 con 0x7f6a78099b80 2024-03-21T08:44:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:11.123 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:44:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 msgr2=0x7f6a54069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 0x7f6a54069fe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a7814fa70 tx=0x7f6a60007d40 comp rx=0 tx=0).stop 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 msgr2=0x7f6a7806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c3c0 tx=0x7f6a6800c880 comp rx=0 tx=0).stop 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 shutdown_connections 2024-03-21T08:44:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6a54067b20 0x7f6a54069fe0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6a7815f260 0x7f6a7814e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6a78099fe0 0x7f6a78157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 --2- 172.21.15.28:0/2148312067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a78099b80 0x7f6a7806d900 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 >> 172.21.15.28:0/2148312067 conn(0x7f6a780b34d0 msgr2=0x7f6a7815d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 shutdown_connections 2024-03-21T08:44:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.121+0000 7f6a7ee00640 1 -- 172.21.15.28:0/2148312067 wait complete. 2024-03-21T08:44:11.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.285+0000 7f5e4c800640 1 Processor -- start 2024-03-21T08:44:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.285+0000 7f5e4c800640 1 -- start start 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e4c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e4415f260 0x7f5e440cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e4c800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5e440cd5d0 0x7f5e44099b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e4c800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e440cdb50 con 0x7f5e4409a0c0 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e4c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e440c5530 con 0x7f5e4415f260 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e4c800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e440cde30 con 0x7f5e440cd5d0 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e4415f260 0x7f5e440cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e4415f260 0x7f5e440cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43746/0 (socket says 172.21.15.28:43746) 2024-03-21T08:44:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 -- 172.21.15.28:0/3007593705 learned_addr learned my addr 172.21.15.28:0/3007593705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 -- 172.21.15.28:0/3007593705 >> v2:172.21.15.151:3300/0 conn(0x7f5e440cd5d0 msgr2=0x7f5e44099b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 --2- 172.21.15.28:0/3007593705 >> v2:172.21.15.151:3300/0 conn(0x7f5e440cd5d0 0x7f5e44099b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 -- 172.21.15.28:0/3007593705 >> v1:172.21.15.151:6789/0 conn(0x7f5e4409a0c0 legacy=0x7f5e440cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.289+0000 7f5e49e00640 1 -- 172.21.15.28:0/3007593705 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440c5780 con 0x7f5e4415f260 2024-03-21T08:44:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e49e00640 1 --2- 172.21.15.28:0/3007593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4415f260 0x7f5e440cd090 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5e440730b0 tx=0x7f5e3801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5281e84654ef15 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e48a00640 1 -- 172.21.15.28:0/3007593705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e380b2070 con 0x7f5e4415f260 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e48a00640 1 -- 172.21.15.28:0/3007593705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e380a9070 con 0x7f5e4415f260 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e48a00640 1 -- 172.21.15.28:0/3007593705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e380ad880 con 0x7f5e4415f260 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 -- 172.21.15.28:0/3007593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4415f260 msgr2=0x7f5e440cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 --2- 172.21.15.28:0/3007593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4415f260 0x7f5e440cd090 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5e440730b0 tx=0x7f5e3801c450 comp rx=0 tx=0).stop 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 -- 172.21.15.28:0/3007593705 shutdown_connections 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 --2- 172.21.15.28:0/3007593705 >> v2:172.21.15.151:3300/0 conn(0x7f5e440cd5d0 0x7f5e44099b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 --2- 172.21.15.28:0/3007593705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4415f260 0x7f5e440cd090 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 -- 172.21.15.28:0/3007593705 >> 172.21.15.28:0/3007593705 conn(0x7f5e440b34d0 msgr2=0x7f5e440b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 -- 172.21.15.28:0/3007593705 shutdown_connections 2024-03-21T08:44:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.293+0000 7f5e4c800640 1 -- 172.21.15.28:0/3007593705 wait complete. 2024-03-21T08:44:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 Processor -- start 2024-03-21T08:44:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- start start 2024-03-21T08:44:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e4409a0c0 0x7f5e44156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e4415f260 0x7f5e4406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44168bc0 con 0x7f5e440cd5d0 2024-03-21T08:44:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44168a40 con 0x7f5e4409a0c0 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44168d40 con 0x7f5e4415f260 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43760/0 (socket says 172.21.15.28:43760) 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 -- 172.21.15.28:0/136018669 learned_addr learned my addr 172.21.15.28:0/136018669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e4409a0c0 0x7f5e44156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4a800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e4415f260 0x7f5e4406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 -- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e4415f260 msgr2=0x7f5e4406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e4415f260 0x7f5e4406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 -- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e4409a0c0 msgr2=0x7f5e44156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e4409a0c0 0x7f5e44156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 -- 172.21.15.28:0/136018669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44156810 con 0x7f5e440cd5d0 2024-03-21T08:44:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e49400640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400b550 tx=0x7f5e3400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34014070 con 0x7f5e440cd5d0 2024-03-21T08:44:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34010890 con 0x7f5e440cd5d0 2024-03-21T08:44:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34010c80 con 0x7f5e440cd5d0 2024-03-21T08:44:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4415c9b0 con 0x7f5e440cd5d0 2024-03-21T08:44:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4415cf70 con 0x7f5e440cd5d0 2024-03-21T08:44:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.297+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4404f410 con 0x7f5e440cd5d0 2024-03-21T08:44:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.301+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5e34017070 con 0x7f5e440cd5d0 2024-03-21T08:44:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.301+0000 7f5e3e000640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 0x7f5e28069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.301+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e34085ea0 con 0x7f5e440cd5d0 2024-03-21T08:44:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.301+0000 7f5e49e00640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 0x7f5e28069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.301+0000 7f5e49e00640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 0x7f5e28069fe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5e380af000 tx=0x7f5e380a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.305+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5e44156810 con 0x7f5e440cd5d0 2024-03-21T08:44:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.469+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e440cbd90 con 0x7f5e440cd5d0 2024-03-21T08:44:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.469+0000 7f5e3e000640 1 -- 172.21.15.28:0/136018669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f5e34010e20 con 0x7f5e440cd5d0 2024-03-21T08:44:11.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:11.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 msgr2=0x7f5e28069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 0x7f5e28069fe0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5e380af000 tx=0x7f5e380a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 msgr2=0x7f5e441562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400b550 tx=0x7f5e3400ba10 comp rx=0 tx=0).stop 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 shutdown_connections 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e28067b20 0x7f5e28069fe0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5e4415f260 0x7f5e4406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e440cd5d0 0x7f5e441562d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 --2- 172.21.15.28:0/136018669 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5e4409a0c0 0x7f5e44156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 >> 172.21.15.28:0/136018669 conn(0x7f5e440b34d0 msgr2=0x7f5e4415e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 shutdown_connections 2024-03-21T08:44:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.473+0000 7f5e4c800640 1 -- 172.21.15.28:0/136018669 wait complete. 2024-03-21T08:44:11.495 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-21T08:44:11.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:44:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 Processor -- start 2024-03-21T08:44:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 -- start start 2024-03-21T08:44:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f385c15de50 0x7f385c162210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0af910 0x7f385c0caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c0cd850 con 0x7f385c0cd0d0 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385c15e230 con 0x7f385c15de50 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f3861c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0cde10 con 0x7f385c0af910 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f385c15de50 0x7f385c162210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f385c15de50 0x7f385c162210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43766/0 (socket says 172.21.15.28:43766) 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 -- 172.21.15.28:0/3277886110 learned_addr learned my addr 172.21.15.28:0/3277886110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385b400640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0af910 0x7f385c0caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 -- 172.21.15.28:0/3277886110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0af910 msgr2=0x7f385c0caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:11.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0af910 0x7f385c0caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 -- 172.21.15.28:0/3277886110 >> v1:172.21.15.151:6789/0 conn(0x7f385c0cd0d0 legacy=0x7f385c0cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.633+0000 7f385aa00640 1 -- 172.21.15.28:0/3277886110 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0cdb30 con 0x7f385c15de50 2024-03-21T08:44:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f385aa00640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c15de50 0x7f385c162210 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f384c009870 tx=0x7f384c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135969f4d9115552 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3859600640 1 -- 172.21.15.28:0/3277886110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0b1070 con 0x7f385c15de50 2024-03-21T08:44:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3859600640 1 -- 172.21.15.28:0/3277886110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c0ac410 con 0x7f385c15de50 2024-03-21T08:44:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3859600640 1 -- 172.21.15.28:0/3277886110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0ac7e0 con 0x7f385c15de50 2024-03-21T08:44:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 -- 172.21.15.28:0/3277886110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c15de50 msgr2=0x7f385c162210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c15de50 0x7f385c162210 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f384c009870 tx=0x7f384c01c4a0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 -- 172.21.15.28:0/3277886110 shutdown_connections 2024-03-21T08:44:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0af910 0x7f385c0caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 --2- 172.21.15.28:0/3277886110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c15de50 0x7f385c162210 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 -- 172.21.15.28:0/3277886110 >> 172.21.15.28:0/3277886110 conn(0x7f385c099a30 msgr2=0x7f385c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 -- 172.21.15.28:0/3277886110 shutdown_connections 2024-03-21T08:44:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.637+0000 7f3861c00640 1 -- 172.21.15.28:0/3277886110 wait complete. 2024-03-21T08:44:11.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 Processor -- start 2024-03-21T08:44:11.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 -- start start 2024-03-21T08:44:11.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0cd0d0 0x7f385c149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f385c15de50 0x7f385c149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0ce2c0 con 0x7f385c0af910 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0ce140 con 0x7f385c15de50 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f3861c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0cb890 con 0x7f385c0cd0d0 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f385aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f385aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43770/0 (socket says 172.21.15.28:43770) 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.641+0000 7f385aa00640 1 -- 172.21.15.28:0/3622159849 learned_addr learned my addr 172.21.15.28:0/3622159849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385a000640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0cd0d0 0x7f385c149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385b400640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f385c15de50 0x7f385c149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 -- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0cd0d0 msgr2=0x7f385c149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0cd0d0 0x7f385c149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 -- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f385c15de50 msgr2=0x7f385c149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f385c15de50 0x7f385c149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 -- 172.21.15.28:0/3622159849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c153080 con 0x7f385c0af910 2024-03-21T08:44:11.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f385aa00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f384c0af040 tx=0x7f384c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0bc070 con 0x7f385c0af910 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c0acd10 con 0x7f385c0af910 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c153360 con 0x7f385c0af910 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0b1040 con 0x7f385c0af910 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c153920 con 0x7f385c0af910 2024-03-21T08:44:11.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c04f410 con 0x7f385c0af910 2024-03-21T08:44:11.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.645+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f384c0bf070 con 0x7f385c0af910 2024-03-21T08:44:11.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.649+0000 7f3852a00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 0x7f383c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:11.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.649+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f384c11f750 con 0x7f385c0af910 2024-03-21T08:44:11.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.649+0000 7f385a000640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 0x7f383c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:11.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.649+0000 7f385a000640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 0x7f383c069fe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f385c0cbe30 tx=0x7f3848009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:11.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.653+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f384c016150 con 0x7f385c0af910 2024-03-21T08:44:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.849+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f385c0cc340 con 0x7f385c0af910 2024-03-21T08:44:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.849+0000 7f3852a00640 1 -- 172.21.15.28:0/3622159849 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f384c0eff30 con 0x7f385c0af910 2024-03-21T08:44:11.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:11.853 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:44:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:44:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 msgr2=0x7f383c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 0x7f383c069fe0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f385c0cbe30 tx=0x7f3848009040 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 msgr2=0x7f385c148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f384c0af040 tx=0x7f384c019240 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 shutdown_connections 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f385c15de50 0x7f385c149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f383c067b20 0x7f383c069fe0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f385c0cd0d0 0x7f385c149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 --2- 172.21.15.28:0/3622159849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f385c0af910 0x7f385c148e90 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 >> 172.21.15.28:0/3622159849 conn(0x7f385c099a30 msgr2=0x7f385c0cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 shutdown_connections 2024-03-21T08:44:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:11.853+0000 7f3861c00640 1 -- 172.21.15.28:0/3622159849 wait complete. 2024-03-21T08:44:11.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:44:12.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 Processor -- start 2024-03-21T08:44:12.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 -- start start 2024-03-21T08:44:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14fc099b80 0x7f14fc15f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 0x7f14fc06c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc0cfac0 con 0x7f14fc0d0880 2024-03-21T08:44:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14fc0cfda0 con 0x7f14fc099b80 2024-03-21T08:44:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f1500a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0d0080 con 0x7f14fc0b0b40 2024-03-21T08:44:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14fc099b80 0x7f14fc15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14fc099b80 0x7f14fc15f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43786/0 (socket says 172.21.15.28:43786) 2024-03-21T08:44:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 -- 172.21.15.28:0/1678196026 learned_addr learned my addr 172.21.15.28:0/1678196026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14fa800640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 0x7f14fc06c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 -- 172.21.15.28:0/1678196026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 msgr2=0x7f14fc06c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 0x7f14fc06c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 -- 172.21.15.28:0/1678196026 >> v1:172.21.15.151:6789/0 conn(0x7f14fc0d0880 legacy=0x7f14fc099f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14f9e00640 1 -- 172.21.15.28:0/1678196026 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc0d0360 con 0x7f14fc099b80 2024-03-21T08:44:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.013+0000 7f14fa800640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 0x7f14fc06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f14f9e00640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc15f220 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8096e6f2e3cf290a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f14f9400640 1 -- 172.21.15.28:0/1678196026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0ab680 con 0x7f14fc099b80 2024-03-21T08:44:12.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f14f9400640 1 -- 172.21.15.28:0/1678196026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0b0070 con 0x7f14fc099b80 2024-03-21T08:44:12.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f14f9400640 1 -- 172.21.15.28:0/1678196026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0aba40 con 0x7f14fc099b80 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 -- 172.21.15.28:0/1678196026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 msgr2=0x7f14fc15f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc15f220 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec01c560 comp rx=0 tx=0).stop 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 -- 172.21.15.28:0/1678196026 shutdown_connections 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0b0b40 0x7f14fc06c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 --2- 172.21.15.28:0/1678196026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc15f220 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 -- 172.21.15.28:0/1678196026 >> 172.21.15.28:0/1678196026 conn(0x7f14fc0b3510 msgr2=0x7f14fc0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 -- 172.21.15.28:0/1678196026 shutdown_connections 2024-03-21T08:44:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.017+0000 7f1500a00640 1 -- 172.21.15.28:0/1678196026 wait complete. 2024-03-21T08:44:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 Processor -- start 2024-03-21T08:44:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 -- start start 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 0x7f14fc0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0d0880 0x7f14fc15d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc06d860 con 0x7f14fc099b80 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0b2d40 con 0x7f14fc0b0b40 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f1500a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc16b050 con 0x7f14fc0d0880 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43798/0 (socket says 172.21.15.28:43798) 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 -- 172.21.15.28:0/3003304050 learned_addr learned my addr 172.21.15.28:0/3003304050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f3e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 0x7f14fc0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14fa800640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0d0880 0x7f14fc15d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 -- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0d0880 msgr2=0x7f14fc15d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0d0880 0x7f14fc15d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 -- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 msgr2=0x7f14fc0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 0x7f14fc0b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f9e00640 1 -- 172.21.15.28:0/3003304050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc155500 con 0x7f14fc099b80 2024-03-21T08:44:12.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.021+0000 7f14f3e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 0x7f14fc0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f9e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0ad040 tx=0x7f14ec01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0be5f0 con 0x7f14fc099b80 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0a7070 con 0x7f14fc099b80 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc0b5090 con 0x7f14fc099b80 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0be9f0 con 0x7f14fc099b80 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc15d190 con 0x7f14fc099b80 2024-03-21T08:44:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc04f410 con 0x7f14fc099b80 2024-03-21T08:44:12.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f14ec0beb90 con 0x7f14fc099b80 2024-03-21T08:44:12.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 0x7f14d8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.025+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f14ec1206e0 con 0x7f14fc099b80 2024-03-21T08:44:12.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.029+0000 7f14f3e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 0x7f14d8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.029+0000 7f14f3e00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 0x7f14d8069fe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f14fc1fa880 tx=0x7f14e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.033+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f14ec0c4060 con 0x7f14fc099b80 2024-03-21T08:44:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.229+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14fc0b43f0 con 0x7f14fc099b80 2024-03-21T08:44:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.229+0000 7f14f2000640 1 -- 172.21.15.28:0/3003304050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f14ec016150 con 0x7f14fc099b80 2024-03-21T08:44:12.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:12.234 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-21T08:43:33.457005+0000","modified":"2024-03-21T08:43:36.469927+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/460337969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":460337969},{"type":"v1","addr":"172.21.15.28:6835","nonce":460337969}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6249,"qdb_cluster":[6249]},"id":3}]} 2024-03-21T08:44:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-03-21T08:44:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 msgr2=0x7f14d8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 0x7f14d8069fe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f14fc1fa880 tx=0x7f14e4009040 comp rx=0 tx=0).stop 2024-03-21T08:44:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 msgr2=0x7f14fc14e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0ad040 tx=0x7f14ec01ada0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 shutdown_connections 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f14d8067b20 0x7f14d8069fe0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f14fc0d0880 0x7f14fc15d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f14fc0b0b40 0x7f14fc0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 --2- 172.21.15.28:0/3003304050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14fc099b80 0x7f14fc14e290 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 >> 172.21.15.28:0/3003304050 conn(0x7f14fc0b3510 msgr2=0x7f14fc0ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 shutdown_connections 2024-03-21T08:44:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.233+0000 7f1500a00640 1 -- 172.21.15.28:0/3003304050 wait complete. 2024-03-21T08:44:12.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.393+0000 7fa1c4a00640 1 Processor -- start 2024-03-21T08:44:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.393+0000 7fa1c4a00640 1 -- start start 2024-03-21T08:44:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1c4a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1c4a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c00cc1d0 con 0x7fa1c00ccc70 2024-03-21T08:44:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1c4a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c00d1730 con 0x7fa1c0099b80 2024-03-21T08:44:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c009a520 con 0x7fa1c015e0c0 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1b5400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:42122/0 (socket says 172.21.15.28:42122) 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1be800640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7fa1c00ccc70 0x7fa1c00cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:54898/0 (socket says 172.21.15.28:54898) 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 -- 172.21.15.28:0/3970644541 learned_addr learned my addr 172.21.15.28:0/3970644541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1b5400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:57056/0 (socket says 172.21.15.28:57056) 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 -- 172.21.15.28:0/3970644541 >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 msgr2=0x7fa1c0099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 --2- 172.21.15.28:0/3970644541 >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 -- 172.21.15.28:0/3970644541 >> v1:172.21.15.151:6789/0 conn(0x7fa1c00ccc70 legacy=0x7fa1c00cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bde00640 1 -- 172.21.15.28:0/3970644541 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c006d450 con 0x7fa1c015e0c0 2024-03-21T08:44:12.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1b5400640 1 --2- 172.21.15.28:0/3970644541 >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:12.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.397+0000 7fa1bd400640 1 -- 172.21.15.28:0/3970644541 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803119023 0 0) 0x7fa1c00cc1d0 con 0x7fa1c00ccc70 2024-03-21T08:44:12.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1bde00640 1 --2- 172.21.15.28:0/3970644541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009870 tx=0x7fa1b001c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1463637e40a4d4a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1bd400640 1 -- 172.21.15.28:0/3970644541 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00a95f0 con 0x7fa1c015e0c0 2024-03-21T08:44:12.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1bd400640 1 -- 172.21.15.28:0/3970644541 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00a9790 con 0x7fa1c015e0c0 2024-03-21T08:44:12.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1bd400640 1 -- 172.21.15.28:0/3970644541 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00a9b60 con 0x7fa1c015e0c0 2024-03-21T08:44:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 -- 172.21.15.28:0/3970644541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 msgr2=0x7fa1c006cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/3970644541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009870 tx=0x7fa1b001c530 comp rx=0 tx=0).stop 2024-03-21T08:44:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 -- 172.21.15.28:0/3970644541 shutdown_connections 2024-03-21T08:44:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/3970644541 >> v2:172.21.15.151:3300/0 conn(0x7fa1c0099b80 0x7fa1c0099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/3970644541 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c015e0c0 0x7fa1c006cf10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 -- 172.21.15.28:0/3970644541 >> 172.21.15.28:0/3970644541 conn(0x7fa1c00b34d0 msgr2=0x7fa1c00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 -- 172.21.15.28:0/3970644541 shutdown_connections 2024-03-21T08:44:12.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.401+0000 7fa1c4a00640 1 -- 172.21.15.28:0/3970644541 wait complete. 2024-03-21T08:44:12.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 Processor -- start 2024-03-21T08:44:12.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- start start 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c00ccc70 0x7fa1c00c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c015e0c0 0x7fa1c01557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c00d10d0 con 0x7fa1c015e0c0 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c015e4a0 con 0x7fa1c00ccc70 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c016ac90 con 0x7fa1c0099b80 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1be800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c015e0c0 0x7fa1c01557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:42136/0 (socket says 172.21.15.28:42136) 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 -- 172.21.15.28:0/2856163749 learned_addr learned my addr 172.21.15.28:0/2856163749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1b5400640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c00ccc70 0x7fa1c00c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 -- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c00ccc70 msgr2=0x7fa1c00c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c00ccc70 0x7fa1c00c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 -- 172.21.15.28:0/2856163749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c015e0c0 msgr2=0x7fa1c01557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c015e0c0 0x7fa1c01557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 -- 172.21.15.28:0/2856163749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c00c7de0 con 0x7fa1c0099b80 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1bde00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa1b001a160 tx=0x7fa1b00192d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00bb640 con 0x7fa1c0099b80 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00bb7e0 con 0x7fa1c0099b80 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c00c80c0 con 0x7fa1c0099b80 2024-03-21T08:44:12.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00bbbd0 con 0x7fa1c0099b80 2024-03-21T08:44:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.405+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c00c8ab0 con 0x7fa1c0099b80 2024-03-21T08:44:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c004f410 con 0x7fa1c0099b80 2024-03-21T08:44:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa1b00c43f0 con 0x7fa1c0099b80 2024-03-21T08:44:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1b6a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 0x7fa19c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa1b0120bf0 con 0x7fa1c0099b80 2024-03-21T08:44:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1b5400640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 0x7fa19c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.409+0000 7fa1b5400640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 0x7fa19c069fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009580 tx=0x7fa1a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.413+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa1b00ad020 con 0x7fa1c0099b80 2024-03-21T08:44:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.593+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1c015ac30 con 0x7fa1c0099b80 2024-03-21T08:44:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1b6a00640 1 -- 172.21.15.28:0/2856163749 <== mon.2 v2:172.21.15.151:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16950 (secure 0 0 0) 0x7fa1b0016150 con 0x7fa1c0099b80 2024-03-21T08:44:12.598 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:12.599 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":54,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:43:35.059619+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-21T08:43:31.890980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-21T08:43:32.446780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 msgr2=0x7fa19c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 0x7fa19c069fe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8009580 tx=0x7fa1a8009190 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 msgr2=0x7fa1c0070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa1b001a160 tx=0x7fa1b00192d0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 shutdown_connections 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c015e0c0 0x7fa1c01557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa1c00ccc70 0x7fa1c00c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa19c067b20 0x7fa19c069fe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 --2- 172.21.15.28:0/2856163749 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa1c0099b80 0x7fa1c0070550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 >> 172.21.15.28:0/2856163749 conn(0x7fa1c00b34d0 msgr2=0x7fa1c015d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 shutdown_connections 2024-03-21T08:44:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.597+0000 7fa1c4a00640 1 -- 172.21.15.28:0/2856163749 wait complete. 2024-03-21T08:44:12.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-21T08:44:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.765+0000 7fbc6f000640 1 Processor -- start 2024-03-21T08:44:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.765+0000 7fbc6f000640 1 -- start start 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc6f000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbc6806b880 0x7fbc680cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc6f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc680cfc20 con 0x7fbc68152cd0 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc6f000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc681604f0 con 0x7fbc6815f750 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc6f000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc681602b0 con 0x7fbc6806b880 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc67e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc68152cd0 0x7fbc6806b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39930/0 (socket says 172.21.15.28:39930) 2024-03-21T08:44:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc67e00640 1 -- 172.21.15.28:0/1928325700 learned_addr learned my addr 172.21.15.28:0/1928325700 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc5f400640 1 --2- 172.21.15.28:0/1928325700 >> v2:172.21.15.151:3300/0 conn(0x7fbc6806b880 0x7fbc680cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345070184 0 0) 0x7fbc680cfc20 con 0x7fbc68152cd0 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c003530 con 0x7fbc68152cd0 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc5f400640 1 -- 172.21.15.28:0/1928325700 >> v1:172.21.15.151:6789/0 conn(0x7fbc6815f750 legacy=0x7fbc6815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748250927 0 0) 0x7fbc4c003530 con 0x7fbc68152cd0 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc5f400640 1 -- 172.21.15.28:0/1928325700 >> v1:172.21.15.28:6789/0 conn(0x7fbc68152cd0 legacy=0x7fbc6806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.769+0000 7fbc5f400640 1 -- 172.21.15.28:0/1928325700 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680cfec0 con 0x7fbc6806b880 2024-03-21T08:44:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc5f400640 1 --2- 172.21.15.28:0/1928325700 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6806b880 0x7fbc680cc4c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097c0 tx=0x7fbc5001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c500339e0c85dee server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc500b0070 con 0x7fbc6806b880 2024-03-21T08:44:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc500ab400 con 0x7fbc6806b880 2024-03-21T08:44:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc67400640 1 -- 172.21.15.28:0/1928325700 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc580016a0 con 0x7fbc6806b880 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- 172.21.15.28:0/1928325700 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6806b880 msgr2=0x7fbc680cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1928325700 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6806b880 0x7fbc680cc4c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097c0 tx=0x7fbc5001c3c0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- 172.21.15.28:0/1928325700 shutdown_connections 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1928325700 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc6806b880 0x7fbc680cc4c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- 172.21.15.28:0/1928325700 >> 172.21.15.28:0/1928325700 conn(0x7fbc68099a30 msgr2=0x7fbc68099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- 172.21.15.28:0/1928325700 shutdown_connections 2024-03-21T08:44:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- 172.21.15.28:0/1928325700 wait complete. 2024-03-21T08:44:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 Processor -- start 2024-03-21T08:44:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.773+0000 7fbc6f000640 1 -- start start 2024-03-21T08:44:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6806b880 0x7fbc68149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc68152cd0 0x7fbc680ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc68169a10 con 0x7fbc6815f750 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc68169890 con 0x7fbc68152cd0 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc68169b90 con 0x7fbc6806b880 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43828/0 (socket says 172.21.15.28:43828) 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 -- 172.21.15.28:0/1194485667 learned_addr learned my addr 172.21.15.28:0/1194485667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc67e00640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6806b880 0x7fbc68149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc5f400640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc68152cd0 0x7fbc680ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 -- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6806b880 msgr2=0x7fbc68149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6806b880 0x7fbc68149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 -- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc68152cd0 msgr2=0x7fbc680ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc68152cd0 0x7fbc680ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 -- 172.21.15.28:0/1194485667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680ad780 con 0x7fbc6815f750 2024-03-21T08:44:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6d000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000c730 tx=0x7fbc6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60003b80 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc60003d20 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6000aa30 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc680ada60 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc680ae020 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.777+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6809ea60 con 0x7fbc6815f750 2024-03-21T08:44:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.781+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbc6000abd0 con 0x7fbc6815f750 2024-03-21T08:44:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.781+0000 7fbc64c00640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.781+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbc60085f90 con 0x7fbc6815f750 2024-03-21T08:44:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.781+0000 7fbc67e00640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.781+0000 7fbc67e00640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc58001ec0 tx=0x7fbc58001d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.785+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbc60056770 con 0x7fbc6815f750 2024-03-21T08:44:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.985+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbc6804f4f0 con 0x7fbc6815f750 2024-03-21T08:44:12.987 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-03-21T08:44:12.985+0000 7f91dfc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-21T08:44:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.985+0000 7fbc64c00640 1 -- 172.21.15.28:0/1194485667 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbc60056070 con 0x7fbc6815f750 2024-03-21T08:44:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-21T08:44:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 msgr2=0x7fbc4406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc58001ec0 tx=0x7fbc58001d10 comp rx=0 tx=0).stop 2024-03-21T08:44:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 msgr2=0x7fbc680c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000c730 tx=0x7fbc6000cbf0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 shutdown_connections 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc6815f750 0x7fbc680c7a20 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbc68152cd0 0x7fbc680ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbc44067b50 0x7fbc4406a010 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 --2- 172.21.15.28:0/1194485667 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbc6806b880 0x7fbc68149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 >> 172.21.15.28:0/1194485667 conn(0x7fbc68099a30 msgr2=0x7fbc68165070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 shutdown_connections 2024-03-21T08:44:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:12.989+0000 7fbc6f000640 1 -- 172.21.15.28:0/1194485667 wait complete. 2024-03-21T08:44:13.005 INFO:tasks.ceph.mds.a.smithi028.stderr: -3151> 2024-03-21T08:43:36.042+0000 7ff168e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-21T08:44:13.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-21T08:44:13.059 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-03-21T08:44:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.157+0000 7f255f200640 1 Processor -- start 2024-03-21T08:44:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.157+0000 7f255f200640 1 -- start start 2024-03-21T08:44:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2558161f90 0x7f255806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25580cfef0 0x7f255815f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255815e2b0 con 0x7f255815ded0 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255815eb70 con 0x7f25580cfef0 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25580d0690 con 0x7f2558161f90 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f255815ded0 0x7f2558161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39954/0 (socket says 172.21.15.28:39954) 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25580cfef0 0x7f255815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255de00640 1 -- 172.21.15.28:0/3823525043 learned_addr learned my addr 172.21.15.28:0/3823525043 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255d400640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2558161f90 0x7f255806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 -- 172.21.15.28:0/3823525043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2558161f90 msgr2=0x7f255806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2558161f90 0x7f255806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 -- 172.21.15.28:0/3823525043 >> v1:172.21.15.28:6789/0 conn(0x7f255815ded0 legacy=0x7f2558161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 -- 172.21.15.28:0/3823525043 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580d03b0 con 0x7f25580cfef0 2024-03-21T08:44:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255ca00640 1 -- 172.21.15.28:0/3823525043 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149406459 0 0) 0x7f255815e2b0 con 0x7f255815ded0 2024-03-21T08:44:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.161+0000 7f255e800640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25580cfef0 0x7f255815f000 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f2554009d90 tx=0x7f255401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1609fd3b7d1038 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255ca00640 1 -- 172.21.15.28:0/3823525043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540b2070 con 0x7f25580cfef0 2024-03-21T08:44:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255ca00640 1 -- 172.21.15.28:0/3823525043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25540a5970 con 0x7f25580cfef0 2024-03-21T08:44:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255ca00640 1 -- 172.21.15.28:0/3823525043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540a5d40 con 0x7f25580cfef0 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- 172.21.15.28:0/3823525043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25580cfef0 msgr2=0x7f255815f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25580cfef0 0x7f255815f000 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f2554009d90 tx=0x7f255401c8b0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- 172.21.15.28:0/3823525043 shutdown_connections 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25580cfef0 0x7f255815f000 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- 172.21.15.28:0/3823525043 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f2558161f90 0x7f255806c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- 172.21.15.28:0/3823525043 >> 172.21.15.28:0/3823525043 conn(0x7f255806fa90 msgr2=0x7f255806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- 172.21.15.28:0/3823525043 shutdown_connections 2024-03-21T08:44:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- 172.21.15.28:0/3823525043 wait complete. 2024-03-21T08:44:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 Processor -- start 2024-03-21T08:44:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- start start 2024-03-21T08:44:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25580cfef0 0x7f255815b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f255815ded0 0x7f25580b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255806d300 con 0x7f2558161f90 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255806d180 con 0x7f255815ded0 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255f200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255806d480 con 0x7f25580cfef0 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43866/0 (socket says 172.21.15.28:43866) 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255e800640 1 -- 172.21.15.28:0/4181382860 learned_addr learned my addr 172.21.15.28:0/4181382860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255de00640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25580cfef0 0x7f255815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.165+0000 7f255d400640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f255815ded0 0x7f25580b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 -- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25580cfef0 msgr2=0x7f255815b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25580cfef0 0x7f255815b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 -- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f255815ded0 msgr2=0x7f25580b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f255815ded0 0x7f25580b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 -- 172.21.15.28:0/4181382860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580ccc60 con 0x7f2558161f90 2024-03-21T08:44:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255e800640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f25540b0040 tx=0x7f2554019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540bd030 con 0x7f2558161f90 2024-03-21T08:44:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25540b8470 con 0x7f2558161f90 2024-03-21T08:44:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25580ccf40 con 0x7f2558161f90 2024-03-21T08:44:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540b2040 con 0x7f2558161f90 2024-03-21T08:44:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25580cd500 con 0x7f2558161f90 2024-03-21T08:44:13.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255804f410 con 0x7f2558161f90 2024-03-21T08:44:13.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f25540c0070 con 0x7f2558161f90 2024-03-21T08:44:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 0x7f253c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.169+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2554120b60 con 0x7f2558161f90 2024-03-21T08:44:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.173+0000 7f255de00640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 0x7f253c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:13.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.173+0000 7f255de00640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 0x7f253c06a010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f254c007fd0 tx=0x7f254c007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:13.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.173+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2554016560 con 0x7f2558161f90 2024-03-21T08:44:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.373+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2558156120 con 0x7f2558161f90 2024-03-21T08:44:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.989+0000 7f2552000640 1 -- 172.21.15.28:0/4181382860 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7f25540f0dc0 con 0x7f2558161f90 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 msgr2=0x7f253c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 0x7f253c06a010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f254c007fd0 tx=0x7f254c007d60 comp rx=0 tx=0).stop 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 msgr2=0x7f25580b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f25540b0040 tx=0x7f2554019650 comp rx=0 tx=0).stop 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 shutdown_connections 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2558161f90 0x7f25580b1510 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f255815ded0 0x7f25580b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f253c067b50 0x7f253c06a010 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 --2- 172.21.15.28:0/4181382860 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f25580cfef0 0x7f255815b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 >> 172.21.15.28:0/4181382860 conn(0x7f255806fa90 msgr2=0x7f2558156d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 shutdown_connections 2024-03-21T08:44:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:13.993+0000 7f255f200640 1 -- 172.21.15.28:0/4181382860 wait complete. 2024-03-21T08:44:14.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-21T08:44:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.165+0000 7f64f4000640 1 Processor -- start 2024-03-21T08:44:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.165+0000 7f64f4000640 1 -- start start 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f4000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 0x7f64ec0af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f4000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f64ec06e030 0x7f64ec0b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f4000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec147b80 con 0x7f64ec1477a0 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f4000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec06e530 con 0x7f64ec06e030 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f4000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec06e810 con 0x7f64ec070600 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f1600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64ec1477a0 0x7f64ec06fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39970/0 (socket says 172.21.15.28:39970) 2024-03-21T08:44:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f1600640 1 -- 172.21.15.28:0/4259839915 learned_addr learned my addr 172.21.15.28:0/4259839915 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f0c00640 1 --2- 172.21.15.28:0/4259839915 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 0x7f64ec0af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f2000640 1 --2- 172.21.15.28:0/4259839915 >> v2:172.21.15.151:3300/0 conn(0x7f64ec06e030 0x7f64ec0b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64ebe00640 1 -- 172.21.15.28:0/4259839915 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677355343 0 0) 0x7f64ec147b80 con 0x7f64ec1477a0 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64ebe00640 1 -- 172.21.15.28:0/4259839915 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d4003530 con 0x7f64ec1477a0 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f0c00640 1 -- 172.21.15.28:0/4259839915 >> v2:172.21.15.151:3300/0 conn(0x7f64ec06e030 msgr2=0x7f64ec0b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f0c00640 1 --2- 172.21.15.28:0/4259839915 >> v2:172.21.15.151:3300/0 conn(0x7f64ec06e030 0x7f64ec0b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f0c00640 1 -- 172.21.15.28:0/4259839915 >> v1:172.21.15.28:6789/0 conn(0x7f64ec1477a0 legacy=0x7f64ec06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.169+0000 7f64f0c00640 1 -- 172.21.15.28:0/4259839915 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec06eaf0 con 0x7f64ec070600 2024-03-21T08:44:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f0c00640 1 --2- 172.21.15.28:0/4259839915 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 0x7f64ec0af000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f64d80097c0 tx=0x7f64d801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f414b34b25da719 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64ebe00640 1 -- 172.21.15.28:0/4259839915 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d80a7070 con 0x7f64ec070600 2024-03-21T08:44:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64ebe00640 1 -- 172.21.15.28:0/4259839915 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d80b0070 con 0x7f64ec070600 2024-03-21T08:44:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64ebe00640 1 -- 172.21.15.28:0/4259839915 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d80ab890 con 0x7f64ec070600 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- 172.21.15.28:0/4259839915 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 msgr2=0x7f64ec0af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- 172.21.15.28:0/4259839915 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 0x7f64ec0af000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f64d80097c0 tx=0x7f64d801c470 comp rx=0 tx=0).stop 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- 172.21.15.28:0/4259839915 shutdown_connections 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- 172.21.15.28:0/4259839915 >> v2:172.21.15.151:3300/0 conn(0x7f64ec06e030 0x7f64ec0b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- 172.21.15.28:0/4259839915 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec070600 0x7f64ec0af000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- 172.21.15.28:0/4259839915 >> 172.21.15.28:0/4259839915 conn(0x7f64ec0b1a70 msgr2=0x7f64ec0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- 172.21.15.28:0/4259839915 shutdown_connections 2024-03-21T08:44:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- 172.21.15.28:0/4259839915 wait complete. 2024-03-21T08:44:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 Processor -- start 2024-03-21T08:44:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- start start 2024-03-21T08:44:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64ec070600 0x7f64ec0af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 0x7f64ec0afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec06f750 con 0x7f64ec06e030 2024-03-21T08:44:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec06f5d0 con 0x7f64ec070600 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.173+0000 7f64f4000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec0b3ac0 con 0x7f64ec1477a0 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43880/0 (socket says 172.21.15.28:43880) 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 -- 172.21.15.28:0/3114746078 learned_addr learned my addr 172.21.15.28:0/3114746078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f0c00640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64ec070600 0x7f64ec0af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f2000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 0x7f64ec0afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 -- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 msgr2=0x7f64ec0afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 0x7f64ec0afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 -- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64ec070600 msgr2=0x7f64ec0af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64ec070600 0x7f64ec0af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 -- 172.21.15.28:0/3114746078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec167730 con 0x7f64ec06e030 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f2000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 0x7f64ec0afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f1600640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0017070 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0005cf0 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00072f0 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec166690 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec166c50 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec068110 con 0x7f64ec06e030 2024-03-21T08:44:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f64e001a070 con 0x7f64ec06e030 2024-03-21T08:44:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.177+0000 7f64e9600640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 0x7f64cc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.181+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f64e0085500 con 0x7f64ec06e030 2024-03-21T08:44:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.181+0000 7f64f0c00640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 0x7f64cc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.181+0000 7f64f0c00640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 0x7f64cc069fe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f64d80af000 tx=0x7f64d80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.181+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f64e0055640 con 0x7f64ec06e030 2024-03-21T08:44:14.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:14.349+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f64ec0acad0 con 0x7f64ec06e030 2024-03-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.197+0000 7f64e9600640 1 -- 172.21.15.28:0/3114746078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) v1 ==== 181+0+0 (secure 0 0 0) 0x7f64e00596e0 con 0x7f64ec06e030 2024-03-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 msgr2=0x7f64cc069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 0x7f64cc069fe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f64d80af000 tx=0x7f64d80a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 msgr2=0x7f64ec15fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 shutdown_connections 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f64cc067b20 0x7f64cc069fe0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f64ec1477a0 0x7f64ec0afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f64ec070600 0x7f64ec0af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 --2- 172.21.15.28:0/3114746078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec06e030 0x7f64ec15fab0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.201+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 >> 172.21.15.28:0/3114746078 conn(0x7f64ec0b1a70 msgr2=0x7f64ec0ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.205+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 shutdown_connections 2024-03-21T08:44:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.205+0000 7f64f4000640 1 -- 172.21.15.28:0/3114746078 wait complete. 2024-03-21T08:44:15.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-21T08:44:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 Processor -- start 2024-03-21T08:44:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 -- start start 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 0x7fd96406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd96415e8f0 0x7fd9640cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9640ce460 con 0x7fd9640722b0 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96406dfa0 con 0x7fd96415e8f0 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd96b400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9640ce6e0 con 0x7fd9640aed60 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd95be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd96415e8f0 0x7fd9640cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd969400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9640722b0 0x7fd9640726d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39974/0 (socket says 172.21.15.28:39974) 2024-03-21T08:44:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd969400640 1 -- 172.21.15.28:0/2614733816 learned_addr learned my addr 172.21.15.28:0/2614733816 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd95be00640 1 -- 172.21.15.28:0/2614733816 >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 msgr2=0x7fd96406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.377+0000 7fd968a00640 1 --2- 172.21.15.28:0/2614733816 >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 0x7fd96406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd963e00640 1 -- 172.21.15.28:0/2614733816 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912329889 0 0) 0x7fd95c0014a0 con 0x7fd9640722b0 2024-03-21T08:44:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd95be00640 1 --2- 172.21.15.28:0/2614733816 >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 0x7fd96406da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd95be00640 1 -- 172.21.15.28:0/2614733816 >> v1:172.21.15.28:6789/0 conn(0x7fd9640722b0 legacy=0x7fd9640726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd95be00640 1 -- 172.21.15.28:0/2614733816 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96406e240 con 0x7fd96415e8f0 2024-03-21T08:44:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd968a00640 1 --2- 172.21.15.28:0/2614733816 >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 0x7fd96406da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd95be00640 1 --2- 172.21.15.28:0/2614733816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96415e8f0 0x7fd9640cbce0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd96406ec00 tx=0x7fd94c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32089786dc431db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd963e00640 1 -- 172.21.15.28:0/2614733816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c0ab600 con 0x7fd96415e8f0 2024-03-21T08:44:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd963e00640 1 -- 172.21.15.28:0/2614733816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c0b0070 con 0x7fd96415e8f0 2024-03-21T08:44:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd963e00640 1 -- 172.21.15.28:0/2614733816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c0ab9c0 con 0x7fd96415e8f0 2024-03-21T08:44:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 -- 172.21.15.28:0/2614733816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96415e8f0 msgr2=0x7fd9640cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 --2- 172.21.15.28:0/2614733816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96415e8f0 0x7fd9640cbce0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd96406ec00 tx=0x7fd94c01c480 comp rx=0 tx=0).stop 2024-03-21T08:44:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 -- 172.21.15.28:0/2614733816 shutdown_connections 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 --2- 172.21.15.28:0/2614733816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96415e8f0 0x7fd9640cbce0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 --2- 172.21.15.28:0/2614733816 >> v2:172.21.15.151:3300/0 conn(0x7fd9640aed60 0x7fd96406da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 -- 172.21.15.28:0/2614733816 >> 172.21.15.28:0/2614733816 conn(0x7fd9640674f0 msgr2=0x7fd96406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 -- 172.21.15.28:0/2614733816 shutdown_connections 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.381+0000 7fd96b400640 1 -- 172.21.15.28:0/2614733816 wait complete. 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 Processor -- start 2024-03-21T08:44:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- start start 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9640aed60 0x7fd964158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd96415e8f0 0x7fd964157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640af2c0 con 0x7fd9640722b0 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640af140 con 0x7fd9640aed60 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640cebe0 con 0x7fd96415e8f0 2024-03-21T08:44:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43904/0 (socket says 172.21.15.28:43904) 2024-03-21T08:44:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 -- 172.21.15.28:0/485573464 learned_addr learned my addr 172.21.15.28:0/485573464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd969400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd96415e8f0 0x7fd964157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd95be00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9640aed60 0x7fd964158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 -- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd96415e8f0 msgr2=0x7fd964157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd96415e8f0 0x7fd964157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 -- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9640aed60 msgr2=0x7fd964158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9640aed60 0x7fd964158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 -- 172.21.15.28:0/485573464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964141ce0 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd968a00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c7f0 tx=0x7fd95400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954007a30 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd954007bd0 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd964141fc0 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9540113f0 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.385+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd964142580 con 0x7fd9640722b0 2024-03-21T08:44:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964068110 con 0x7fd9640722b0 2024-03-21T08:44:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd954011590 con 0x7fd9640722b0 2024-03-21T08:44:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd961600640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 0x7fd940069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fd954086080 con 0x7fd9640722b0 2024-03-21T08:44:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd95be00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 0x7fd940069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:15.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.389+0000 7fd95be00640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 0x7fd940069fe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0af000 tx=0x7fd94c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.393+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd954015020 con 0x7fd9640722b0 2024-03-21T08:44:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:15.561+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd9640cd4e0 con 0x7fd9640722b0 2024-03-21T08:44:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.209+0000 7fd961600640 1 -- 172.21.15.28:0/485573464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v57) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd954056400 con 0x7fd9640722b0 2024-03-21T08:44:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-03-21T08:44:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 msgr2=0x7fd940069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 0x7fd940069fe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0af000 tx=0x7fd94c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 msgr2=0x7fd96413aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c7f0 tx=0x7fd95400ccb0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 shutdown_connections 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd940067b20 0x7fd940069fe0 secure :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0af000 tx=0x7fd94c0a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd96415e8f0 0x7fd964157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd9640aed60 0x7fd964158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 --2- 172.21.15.28:0/485573464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9640722b0 0x7fd96413aa70 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 >> 172.21.15.28:0/485573464 conn(0x7fd9640674f0 msgr2=0x7fd96415ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 shutdown_connections 2024-03-21T08:44:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.213+0000 7fd96b400640 1 -- 172.21.15.28:0/485573464 wait complete. 2024-03-21T08:44:16.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-21T08:44:16.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.381+0000 7faedcc00640 1 Processor -- start 2024-03-21T08:44:16.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.381+0000 7faedcc00640 1 -- start start 2024-03-21T08:44:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faedcc00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faed80b4800 0x7faed80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faedcc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed80cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faedcc00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80cd670 con 0x7faed80b4420 2024-03-21T08:44:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faedcc00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed815bf90 con 0x7faed80b4800 2024-03-21T08:44:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faedcc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed815b1d0 con 0x7faed8072190 2024-03-21T08:44:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faed6800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed80cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7faed80b4800 0x7faed80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faed5e00640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7faed80b4420 0x7faed8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:54920/0 (socket says 172.21.15.28:54920) 2024-03-21T08:44:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faed6800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed80cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:42208/0 (socket says 172.21.15.28:42208) 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faed5e00640 1 -- 172.21.15.28:0/1141274857 learned_addr learned my addr 172.21.15.28:0/1141274857 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 --2- 172.21.15.28:0/1141274857 >> v2:172.21.15.151:3300/0 conn(0x7faed80b4800 0x7faed80ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:57130/0 (socket says 172.21.15.28:57130) 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 -- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 msgr2=0x7faed80cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 --2- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed80cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 -- 172.21.15.28:0/1141274857 >> v1:172.21.15.151:6789/0 conn(0x7faed80b4420 legacy=0x7faed8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faed5400640 1 -- 172.21.15.28:0/1141274857 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228300499 0 0) 0x7faed80cd670 con 0x7faed80b4420 2024-03-21T08:44:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.385+0000 7faecd400640 1 -- 172.21.15.28:0/1141274857 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed815b4b0 con 0x7faed80b4800 2024-03-21T08:44:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faecd400640 1 --2- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4800 0x7faed80ccbf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faec00097c0 tx=0x7faec001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d21ade5e7babf15 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faed5400640 1 -- 172.21.15.28:0/1141274857 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec00a7070 con 0x7faed80b4800 2024-03-21T08:44:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faed5400640 1 -- 172.21.15.28:0/1141274857 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec00b0070 con 0x7faed80b4800 2024-03-21T08:44:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faed5400640 1 -- 172.21.15.28:0/1141274857 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec00ab890 con 0x7faed80b4800 2024-03-21T08:44:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4800 msgr2=0x7faed80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4800 0x7faed80ccbf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faec00097c0 tx=0x7faec001c470 comp rx=0 tx=0).stop 2024-03-21T08:44:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- 172.21.15.28:0/1141274857 shutdown_connections 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed80cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- 172.21.15.28:0/1141274857 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4800 0x7faed80ccbf0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- 172.21.15.28:0/1141274857 >> 172.21.15.28:0/1141274857 conn(0x7faed80674f0 msgr2=0x7faed806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- 172.21.15.28:0/1141274857 shutdown_connections 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- 172.21.15.28:0/1141274857 wait complete. 2024-03-21T08:44:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 Processor -- start 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- start start 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 0x7faed8142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80b3460 con 0x7faed8154580 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80b32e0 con 0x7faed80b4420 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.389+0000 7faedcc00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed80b35e0 con 0x7faed8072190 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43908/0 (socket says 172.21.15.28:43908) 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 -- 172.21.15.28:0/1128212391 learned_addr learned my addr 172.21.15.28:0/1128212391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecd400640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 0x7faed8142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed5e00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 -- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 msgr2=0x7faed8154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed8154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 -- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 msgr2=0x7faed8142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 0x7faed8142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 -- 172.21.15.28:0/1128212391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8152fa0 con 0x7faed8154580 2024-03-21T08:44:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecd400640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 0x7faed8142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faed6800640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7faed000efc0 tx=0x7faed000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0018020 con 0x7faed8154580 2024-03-21T08:44:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed000cd40 con 0x7faed8154580 2024-03-21T08:44:16.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed8153280 con 0x7faed8154580 2024-03-21T08:44:16.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00146c0 con 0x7faed8154580 2024-03-21T08:44:16.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed8153840 con 0x7faed8154580 2024-03-21T08:44:16.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed8073430 con 0x7faed8154580 2024-03-21T08:44:16.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.393+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7faed001b020 con 0x7faed8154580 2024-03-21T08:44:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.397+0000 7faecea00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 0x7faeb4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.397+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7faed00870a0 con 0x7faed8154580 2024-03-21T08:44:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.397+0000 7faed5e00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 0x7faeb4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.397+0000 7faed5e00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 0x7faeb4069fe0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faec8000f70 tx=0x7faec8007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.401+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faed0022020 con 0x7faed8154580 2024-03-21T08:44:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.561+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7faed8165580 con 0x7faed8154580 2024-03-21T08:44:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.565+0000 7faecea00640 1 -- 172.21.15.28:0/1128212391 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7faed0010040 con 0x7faed8154580 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 msgr2=0x7faeb4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 0x7faeb4069fe0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faec8000f70 tx=0x7faec8007b60 comp rx=0 tx=0).stop 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 msgr2=0x7faed8143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7faed000efc0 tx=0x7faed000c490 comp rx=0 tx=0).stop 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 shutdown_connections 2024-03-21T08:44:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8154580 0x7faed8143470 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7faed80b4420 0x7faed8142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7faeb4067b20 0x7faeb4069fe0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 --2- 172.21.15.28:0/1128212391 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7faed8072190 0x7faed8154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 >> 172.21.15.28:0/1128212391 conn(0x7faed80674f0 msgr2=0x7faed8159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 shutdown_connections 2024-03-21T08:44:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.569+0000 7faedcc00640 1 -- 172.21.15.28:0/1128212391 wait complete. 2024-03-21T08:44:16.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-21T08:44:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 Processor -- start 2024-03-21T08:44:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 -- start start 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 0x7fd4ac099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0d1730 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac09a520 con 0x7fd4ac099b80 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4b1c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cc1d0 con 0x7fd4ac15e0c0 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43912/0 (socket says 172.21.15.28:43912) 2024-03-21T08:44:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 -- 172.21.15.28:0/856006658 learned_addr learned my addr 172.21.15.28:0/856006658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/856006658 >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 0x7fd4ac099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 -- 172.21.15.28:0/856006658 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 msgr2=0x7fd4ac06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac06cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 -- 172.21.15.28:0/856006658 >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 msgr2=0x7fd4ac099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 --2- 172.21.15.28:0/856006658 >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 0x7fd4ac099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4ab400640 1 -- 172.21.15.28:0/856006658 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac06d450 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/856006658 >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 0x7fd4ac099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.733+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac06cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4ab400640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac068d60 tx=0x7fd4a401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55658a5ba529777 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4aa000640 1 -- 172.21.15.28:0/856006658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40a5840 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4aa000640 1 -- 172.21.15.28:0/856006658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a40a59e0 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4aa000640 1 -- 172.21.15.28:0/856006658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40a5db0 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 -- 172.21.15.28:0/856006658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac0ccc70 msgr2=0x7fd4ac0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac068d60 tx=0x7fd4a401c9c0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 -- 172.21.15.28:0/856006658 shutdown_connections 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac0cd100 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/856006658 >> v2:172.21.15.151:3300/0 conn(0x7fd4ac099b80 0x7fd4ac099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/856006658 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac06cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 -- 172.21.15.28:0/856006658 >> 172.21.15.28:0/856006658 conn(0x7fd4ac0b34d0 msgr2=0x7fd4ac0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 -- 172.21.15.28:0/856006658 shutdown_connections 2024-03-21T08:44:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.737+0000 7fd4b1c00640 1 -- 172.21.15.28:0/856006658 wait complete. 2024-03-21T08:44:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 Processor -- start 2024-03-21T08:44:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- start start 2024-03-21T08:44:16.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac0b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cfcb0 con 0x7fd4ac099b80 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac15e4a0 con 0x7fd4ac0ccc70 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cfe30 con 0x7fd4ac15e0c0 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43928/0 (socket says 172.21.15.28:43928) 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 -- 172.21.15.28:0/2331204495 learned_addr learned my addr 172.21.15.28:0/2331204495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4ab400640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac0b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 -- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 msgr2=0x7fd4ac0b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac0b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 -- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4ac0ccc70 msgr2=0x7fd4ac161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:16.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 -- 172.21.15.28:0/2331204495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0b71b0 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4aaa00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c7f0 tx=0x7fd49c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c015070 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c007820 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c007c10 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac0b7490 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac157a70 con 0x7fd4ac099b80 2024-03-21T08:44:16.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.741+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac0b7660 con 0x7fd4ac099b80 2024-03-21T08:44:16.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.745+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd49c018070 con 0x7fd4ac099b80 2024-03-21T08:44:16.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.745+0000 7fd4a2a00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 0x7fd488069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:16.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.745+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd49c086050 con 0x7fd4ac099b80 2024-03-21T08:44:16.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.745+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 0x7fd488069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.745+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 0x7fd488069fe0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac1fa4b0 tx=0x7fd494008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.749+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd49c056aa0 con 0x7fd4ac099b80 2024-03-21T08:44:16.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:16.973+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7fd4ac04f410 con 0x7fd4ac099b80 2024-03-21T08:44:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.209+0000 7fd4a2a00640 1 -- 172.21.15.28:0/2331204495 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v195) v1 ==== 134+0+0 (secure 0 0 0) 0x7fd49c0563a0 con 0x7fd4ac099b80 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 msgr2=0x7fd488069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 0x7fd488069fe0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac1fa4b0 tx=0x7fd494008040 comp rx=0 tx=0).stop 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 msgr2=0x7fd4ac160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c7f0 tx=0x7fd49c00ccb0 comp rx=0 tx=0).stop 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 shutdown_connections 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd488067b20 0x7fd488069fe0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd4ac15e0c0 0x7fd4ac0b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4ac0ccc70 0x7fd4ac161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 --2- 172.21.15.28:0/2331204495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac099b80 0x7fd4ac160c50 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 >> 172.21.15.28:0/2331204495 conn(0x7fd4ac0b34d0 msgr2=0x7fd4ac0c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 shutdown_connections 2024-03-21T08:44:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:17.213+0000 7fd4b1c00640 1 -- 172.21.15.28:0/2331204495 wait complete. 2024-03-21T08:44:17.232 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-21T08:44:17.232 INFO:tasks.cephfs_test_runner: 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner: 2024-03-21T08:44:17.233 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:Ran 2 tests in 143.956s 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-21T08:44:17.234 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-21T08:44:17.235 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-21T08:44:17.235 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-21T08:44:17.235 INFO:tasks.cephfs_test_runner: 2024-03-21T08:44:17.235 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:44:17.483 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=75db766259d14ced947da6222804275b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_2edfee68ad7243bed422a049b6f5f58619300f11/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-21T08:44:17.484 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-21T08:44:17.493 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-21T08:44:17.523 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-21T08:44:17.523 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:44:17.523 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:17.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:44:17.538 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-21T08:44:17.538 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:17.587 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357712k,nr_inodes=4089428,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7543 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-21T08:44:17.587 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:44:17.587 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:17.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:44:17.597 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-21T08:44:17.597 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-21T08:44:17.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354624k,nr_inodes=4088656,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1455 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275376k,nr_inodes=818844,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-21T08:44:17.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:17.650 DEBUG:teuthology.orchestra.run.smithi151:> ip netns list 2024-03-21T08:44:17.702 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-21T08:44:17.702 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-21T08:44:17.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:17.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-21T08:44:17.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:17.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-21T08:44:17.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:17.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo ip link delete ceph-brx 2024-03-21T08:44:18.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:18.028 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-21T08:44:18.038 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-21T08:44:18.039 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-21T08:44:18.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:18.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-21T08:44:18.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:18.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-21T08:44:18.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-21T08:44:18.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-03-21T08:44:18.385 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-21T08:44:18.395 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-21T08:44:18.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:44:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.540+0000 7fb373600640 1 Processor -- start 2024-03-21T08:44:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.540+0000 7fb373600640 1 -- start start 2024-03-21T08:44:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c162030 0x7fb36c0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb36c162410 0x7fb36c0ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fb36c071980 0x7fb36c06e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0af2c0 con 0x7fb36c162410 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0ce690 con 0x7fb36c071980 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb373600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0cda80 con 0x7fb36c162030 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb36c162410 0x7fb36c0ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb36c162410 0x7fb36c0ceb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43942/0 (socket says 172.21.15.28:43942) 2024-03-21T08:44:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 -- 172.21.15.28:0/3884817122 learned_addr learned my addr 172.21.15.28:0/3884817122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb371600640 1 --2- 172.21.15.28:0/3884817122 >> v2:172.21.15.151:3300/0 conn(0x7fb36c071980 0x7fb36c06e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb370c00640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c162030 0x7fb36c0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 -- 172.21.15.28:0/3884817122 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c162030 msgr2=0x7fb36c0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c162030 0x7fb36c0cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 -- 172.21.15.28:0/3884817122 >> v2:172.21.15.151:3300/0 conn(0x7fb36c071980 msgr2=0x7fb36c06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 --2- 172.21.15.28:0/3884817122 >> v2:172.21.15.151:3300/0 conn(0x7fb36c071980 0x7fb36c06e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 -- 172.21.15.28:0/3884817122 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0cdd60 con 0x7fb36c162410 2024-03-21T08:44:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.544+0000 7fb363e00640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c0ceb20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0b2570 tx=0x7fb35401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb536174b0fc8cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb36be00640 1 -- 172.21.15.28:0/3884817122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3540ab5f0 con 0x7fb36c162410 2024-03-21T08:44:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb36be00640 1 -- 172.21.15.28:0/3884817122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3540b0070 con 0x7fb36c162410 2024-03-21T08:44:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb36be00640 1 -- 172.21.15.28:0/3884817122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3540ab9b0 con 0x7fb36c162410 2024-03-21T08:44:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- 172.21.15.28:0/3884817122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 msgr2=0x7fb36c0ceb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c0ceb20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0b2570 tx=0x7fb35401c520 comp rx=0 tx=0).stop 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- 172.21.15.28:0/3884817122 shutdown_connections 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- 172.21.15.28:0/3884817122 >> v2:172.21.15.151:3300/0 conn(0x7fb36c071980 0x7fb36c06e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c0ceb20 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- 172.21.15.28:0/3884817122 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c162030 0x7fb36c0cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- 172.21.15.28:0/3884817122 >> 172.21.15.28:0/3884817122 conn(0x7fb36c0cc230 msgr2=0x7fb36c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- 172.21.15.28:0/3884817122 shutdown_connections 2024-03-21T08:44:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- 172.21.15.28:0/3884817122 wait complete. 2024-03-21T08:44:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 Processor -- start 2024-03-21T08:44:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- start start 2024-03-21T08:44:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c071980 0x7fb36c15d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb36c162030 0x7fb36c152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0b3600 con 0x7fb36c162410 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0b3480 con 0x7fb36c162030 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb373600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0b3780 con 0x7fb36c071980 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb371600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb371600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43952/0 (socket says 172.21.15.28:43952) 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb371600640 1 -- 172.21.15.28:0/3969310965 learned_addr learned my addr 172.21.15.28:0/3969310965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb370c00640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c071980 0x7fb36c15d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.548+0000 7fb363e00640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb36c162030 0x7fb36c152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 -- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c071980 msgr2=0x7fb36c15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c071980 0x7fb36c15d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 -- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb36c162030 msgr2=0x7fb36c152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb36c162030 0x7fb36c152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 -- 172.21.15.28:0/3969310965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c162cd0 con 0x7fb36c162410 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb371600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb364007e90 tx=0x7fb36400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364014420 con 0x7fb36c162410 2024-03-21T08:44:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364019070 con 0x7fb36c162410 2024-03-21T08:44:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c164960 con 0x7fb36c162410 2024-03-21T08:44:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364014820 con 0x7fb36c162410 2024-03-21T08:44:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c164f20 con 0x7fb36c162410 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c04f410 con 0x7fb36c162410 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fb3640149c0 con 0x7fb36c162410 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 0x7fb348069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.552+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fb364086560 con 0x7fb36c162410 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.556+0000 7fb370c00640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 0x7fb348069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.556+0000 7fb370c00640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 0x7fb348069fe0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb35c007a60 tx=0x7fb35c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.560+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3640569f0 con 0x7fb36c162410 2024-03-21T08:44:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.720+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb36c0ccfd0 con 0x7fb348067b20 2024-03-21T08:44:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.720+0000 7fb369600640 1 -- 172.21.15.28:0/3969310965 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13463 (secure 0 0 0) 0x7fb36c0ccfd0 con 0x7fb348067b20 2024-03-21T08:44:18.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:44:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 msgr2=0x7fb348069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 0x7fb348069fe0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb35c007a60 tx=0x7fb35c008040 comp rx=0 tx=0).stop 2024-03-21T08:44:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 msgr2=0x7fb36c15ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb364007e90 tx=0x7fb36400e4b0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 shutdown_connections 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c162410 0x7fb36c15ac80 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fb36c162030 0x7fb36c152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fb348067b20 0x7fb348069fe0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 --2- 172.21.15.28:0/3969310965 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fb36c071980 0x7fb36c15d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 >> 172.21.15.28:0/3969310965 conn(0x7fb36c0cc230 msgr2=0x7fb36c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 shutdown_connections 2024-03-21T08:44:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.724+0000 7fb373600640 1 -- 172.21.15.28:0/3969310965 wait complete. 2024-03-21T08:44:18.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-21T08:44:17.764530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239076,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735644,"statfs":{"total":773094113280,"available":772849299456,"internally_reserved":0,"allocated":6438912,"data_stored":3544028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001691"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-21T08:44:12.995621+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:44:12.995621+0000","last_peered":"2024-03-21T08:44:12.995621+0000","last_clean":"2024-03-21T08:44:12.995621+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:44:12.995621+0000","last_undegraded":"2024-03-21T08:44:12.995621+0000","last_fullsized":"2024-03-21T08:44:12.995621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":884902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":638976,"data_stored":287901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":659456,"data_stored":294430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":295026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":304199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1261568,"data_stored":895149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:44:18.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:44:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 Processor -- start 2024-03-21T08:44:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 -- start start 2024-03-21T08:44:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6dc80b5520 0x7f6dc80cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc815bf00 con 0x7f6dc80af060 2024-03-21T08:44:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc80cdad0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dce000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc80ce6e0 con 0x7f6dc80b5520 2024-03-21T08:44:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dbea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6dc80b5520 0x7f6dc80cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dc7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6dc80af060 0x7f6dc80b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39990/0 (socket says 172.21.15.28:39990) 2024-03-21T08:44:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dc7400640 1 -- 172.21.15.28:0/3525140918 learned_addr learned my addr 172.21.15.28:0/3525140918 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dbea00640 1 -- 172.21.15.28:0/3525140918 >> v1:172.21.15.151:6789/0 conn(0x7f6dc80b34f0 legacy=0x7f6dc815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dbea00640 1 -- 172.21.15.28:0/3525140918 >> v1:172.21.15.28:6789/0 conn(0x7f6dc80af060 legacy=0x7f6dc80b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dbea00640 1 -- 172.21.15.28:0/3525140918 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc80cddb0 con 0x7f6dc80b5520 2024-03-21T08:44:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.884+0000 7f6dc6a00640 1 -- 172.21.15.28:0/3525140918 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939422445 0 0) 0x7f6dc815bf00 con 0x7f6dc80af060 2024-03-21T08:44:18.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dbea00640 1 --2- 172.21.15.28:0/3525140918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b5520 0x7f6dc80cd380 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80ce860 tx=0x7f6db001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d52b0fa4a92217 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dc6a00640 1 -- 172.21.15.28:0/3525140918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00b0070 con 0x7f6dc80b5520 2024-03-21T08:44:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dc6a00640 1 -- 172.21.15.28:0/3525140918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db00ab600 con 0x7f6dc80b5520 2024-03-21T08:44:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dc6a00640 1 -- 172.21.15.28:0/3525140918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00ab9d0 con 0x7f6dc80b5520 2024-03-21T08:44:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 -- 172.21.15.28:0/3525140918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b5520 msgr2=0x7f6dc80cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 --2- 172.21.15.28:0/3525140918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b5520 0x7f6dc80cd380 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80ce860 tx=0x7f6db001c410 comp rx=0 tx=0).stop 2024-03-21T08:44:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 -- 172.21.15.28:0/3525140918 shutdown_connections 2024-03-21T08:44:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 --2- 172.21.15.28:0/3525140918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b5520 0x7f6dc80cd380 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 -- 172.21.15.28:0/3525140918 >> 172.21.15.28:0/3525140918 conn(0x7f6dc80cc230 msgr2=0x7f6dc80cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 -- 172.21.15.28:0/3525140918 shutdown_connections 2024-03-21T08:44:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.888+0000 7f6dce000640 1 -- 172.21.15.28:0/3525140918 wait complete. 2024-03-21T08:44:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 Processor -- start 2024-03-21T08:44:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- start start 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6dc80b5520 0x7f6dc81f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 0x7f6dc8156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc815d2a0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc815d120 con 0x7f6dc8150820 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc815d420 con 0x7f6dc80b5520 2024-03-21T08:44:18.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43972/0 (socket says 172.21.15.28:43972) 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 -- 172.21.15.28:0/2278525981 learned_addr learned my addr 172.21.15.28:0/2278525981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dbea00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6dc80b5520 0x7f6dc81f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7e00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 0x7f6dc8156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 -- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6dc80b5520 msgr2=0x7f6dc81f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6dc80b5520 0x7f6dc81f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 -- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 msgr2=0x7f6dc8156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 0x7f6dc8156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 -- 172.21.15.28:0/2278525981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc81f79a0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7e00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 0x7f6dc8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c4a0 tx=0x7f6db800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8018070 con 0x7f6dc80b34f0 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db80074e0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db8004a30 con 0x7f6dc80b34f0 2024-03-21T08:44:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc8155f00 con 0x7f6dc80b34f0 2024-03-21T08:44:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.892+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc81564c0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc804f410 con 0x7f6dc80b34f0 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f6db8004bd0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dbfe00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 0x7f6da4069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6db80851d0 con 0x7f6dc80b34f0 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dbea00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 0x7f6da4069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.896+0000 7f6dbea00640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 0x7f6da4069fe0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6db00ad000 tx=0x7f6db0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:18.900+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6db8055cb0 con 0x7f6dc80b34f0 2024-03-21T08:44:19.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.060+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6dc80b1250 con 0x7f6da4067b20 2024-03-21T08:44:19.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.060+0000 7f6dbfe00640 1 -- 172.21.15.28:0/2278525981 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13463 (secure 0 0 0) 0x7f6dc80b1250 con 0x7f6da4067b20 2024-03-21T08:44:19.061 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:19.062 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:44:19.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 msgr2=0x7f6da4069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 0x7f6da4069fe0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6db00ad000 tx=0x7f6db0008040 comp rx=0 tx=0).stop 2024-03-21T08:44:19.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 msgr2=0x7f6dc81514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c4a0 tx=0x7f6db800c960 comp rx=0 tx=0).stop 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 shutdown_connections 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6dc8150820 0x7f6dc8156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f6da4067b20 0x7f6da4069fe0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f6dc80b5520 0x7f6dc81f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 --2- 172.21.15.28:0/2278525981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc80b34f0 0x7f6dc81514d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 >> 172.21.15.28:0/2278525981 conn(0x7f6dc80cc230 msgr2=0x7f6dc8073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 shutdown_connections 2024-03-21T08:44:19.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.064+0000 7f6dce000640 1 -- 172.21.15.28:0/2278525981 wait complete. 2024-03-21T08:44:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-21T08:44:17.764530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239076,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735644,"statfs":{"total":773094113280,"available":772849299456,"internally_reserved":0,"allocated":6438912,"data_stored":3544028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001691"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-21T08:44:12.995621+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:44:12.995621+0000","last_peered":"2024-03-21T08:44:12.995621+0000","last_clean":"2024-03-21T08:44:12.995621+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:44:12.995621+0000","last_undegraded":"2024-03-21T08:44:12.995621+0000","last_fullsized":"2024-03-21T08:44:12.995621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":884902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":638976,"data_stored":287901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":659456,"data_stored":294430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":295026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":304199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1261568,"data_stored":895149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:44:19.082 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-21T08:44:19.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:44:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.224+0000 7f5444400640 1 Processor -- start 2024-03-21T08:44:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.224+0000 7f5444400640 1 -- start start 2024-03-21T08:44:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5444400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 0x7f543c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5444400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f543c06b880 0x7f543c161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5444400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c0d0540 con 0x7f543c0aff70 2024-03-21T08:44:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5444400640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c0cd6e0 con 0x7f543c06b880 2024-03-21T08:44:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5444400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0cd9c0 con 0x7f543c15dd00 2024-03-21T08:44:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f543c0aff70 0x7f543c06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39998/0 (socket says 172.21.15.28:39998) 2024-03-21T08:44:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441000640 1 -- 172.21.15.28:0/1839615377 learned_addr learned my addr 172.21.15.28:0/1839615377 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441a00640 1 --2- 172.21.15.28:0/1839615377 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 0x7f543c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5442400640 1 --2- 172.21.15.28:0/1839615377 >> v2:172.21.15.151:3300/0 conn(0x7f543c06b880 0x7f543c161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f543be00640 1 -- 172.21.15.28:0/1839615377 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101990754 0 0) 0x7f543c0d0540 con 0x7f543c0aff70 2024-03-21T08:44:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f543be00640 1 -- 172.21.15.28:0/1839615377 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424003530 con 0x7f543c0aff70 2024-03-21T08:44:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441a00640 1 -- 172.21.15.28:0/1839615377 >> v2:172.21.15.151:3300/0 conn(0x7f543c06b880 msgr2=0x7f543c161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441a00640 1 --2- 172.21.15.28:0/1839615377 >> v2:172.21.15.151:3300/0 conn(0x7f543c06b880 0x7f543c161a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441a00640 1 -- 172.21.15.28:0/1839615377 >> v1:172.21.15.28:6789/0 conn(0x7f543c0aff70 legacy=0x7f543c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.228+0000 7f5441a00640 1 -- 172.21.15.28:0/1839615377 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0cdca0 con 0x7f543c15dd00 2024-03-21T08:44:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5441a00640 1 --2- 172.21.15.28:0/1839615377 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 0x7f543c0cfaf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54300097c0 tx=0x7f543001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3e9db9ead129f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f543be00640 1 -- 172.21.15.28:0/1839615377 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54300aa5f0 con 0x7f543c15dd00 2024-03-21T08:44:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f543be00640 1 -- 172.21.15.28:0/1839615377 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54300af070 con 0x7f543c15dd00 2024-03-21T08:44:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f543be00640 1 -- 172.21.15.28:0/1839615377 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54300aa990 con 0x7f543c15dd00 2024-03-21T08:44:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- 172.21.15.28:0/1839615377 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 msgr2=0x7f543c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 --2- 172.21.15.28:0/1839615377 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 0x7f543c0cfaf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54300097c0 tx=0x7f543001c470 comp rx=0 tx=0).stop 2024-03-21T08:44:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- 172.21.15.28:0/1839615377 shutdown_connections 2024-03-21T08:44:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 --2- 172.21.15.28:0/1839615377 >> v2:172.21.15.151:3300/0 conn(0x7f543c06b880 0x7f543c161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 --2- 172.21.15.28:0/1839615377 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c15dd00 0x7f543c0cfaf0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- 172.21.15.28:0/1839615377 >> 172.21.15.28:0/1839615377 conn(0x7f543c099a30 msgr2=0x7f543c099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:19.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- 172.21.15.28:0/1839615377 shutdown_connections 2024-03-21T08:44:19.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- 172.21.15.28:0/1839615377 wait complete. 2024-03-21T08:44:19.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 Processor -- start 2024-03-21T08:44:19.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.232+0000 7f5444400640 1 -- start start 2024-03-21T08:44:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 0x7f543c155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c0aff70 0x7f543c15f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c16ae60 con 0x7f543c15dd00 2024-03-21T08:44:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c16ace0 con 0x7f543c06b880 2024-03-21T08:44:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c16afe0 con 0x7f543c0aff70 2024-03-21T08:44:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43986/0 (socket says 172.21.15.28:43986) 2024-03-21T08:44:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 -- 172.21.15.28:0/733073553 learned_addr learned my addr 172.21.15.28:0/733073553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5441000640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c0aff70 0x7f543c15f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5441a00640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 0x7f543c155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 -- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c0aff70 msgr2=0x7f543c15f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c0aff70 0x7f543c15f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 -- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 msgr2=0x7f543c155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 0x7f543c155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 -- 172.21.15.28:0/733073553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c161320 con 0x7f543c15dd00 2024-03-21T08:44:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5441a00640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 0x7f543c155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5442400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f543400efc0 tx=0x7f543400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434018020 con 0x7f543c15dd00 2024-03-21T08:44:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5434014480 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c156130 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434014870 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c1569d0 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c04f410 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.236+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f543401b020 con 0x7f543c15dd00 2024-03-21T08:44:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.240+0000 7f5439600640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 0x7f541c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.240+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5434086d80 con 0x7f543c15dd00 2024-03-21T08:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.240+0000 7f5441a00640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 0x7f541c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.240+0000 7f5441a00640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 0x7f541c069fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54300ae000 tx=0x7f54300a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.244+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5434022030 con 0x7f543c15dd00 2024-03-21T08:44:19.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.400+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f543c0cd4c0 con 0x7f541c067b20 2024-03-21T08:44:19.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5439600640 1 -- 172.21.15.28:0/733073553 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13463 (secure 0 0 0) 0x7f543c0cd4c0 con 0x7f541c067b20 2024-03-21T08:44:19.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:19.404 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:44:19.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 msgr2=0x7f541c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 0x7f541c069fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f54300ae000 tx=0x7f54300a7000 comp rx=0 tx=0).stop 2024-03-21T08:44:19.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 msgr2=0x7f543c1571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f543400efc0 tx=0x7f543400c490 comp rx=0 tx=0).stop 2024-03-21T08:44:19.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 shutdown_connections 2024-03-21T08:44:19.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f543c15dd00 0x7f543c1571d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f541c067b20 0x7f541c069fe0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f543c0aff70 0x7f543c15f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 --2- 172.21.15.28:0/733073553 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f543c06b880 0x7f543c155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 >> 172.21.15.28:0/733073553 conn(0x7f543c099a30 msgr2=0x7f543c0af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:19.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 shutdown_connections 2024-03-21T08:44:19.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.404+0000 7f5444400640 1 -- 172.21.15.28:0/733073553 wait complete. 2024-03-21T08:44:19.424 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-21T08:44:17.764530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239076,"kb_used_data":6288,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735644,"statfs":{"total":773094113280,"available":772849299456,"internally_reserved":0,"allocated":6438912,"data_stored":3544028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001691"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-21T08:44:12.995621+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:44:12.995621+0000","last_peered":"2024-03-21T08:44:12.995621+0000","last_clean":"2024-03-21T08:44:12.995621+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:44:12.995621+0000","last_undegraded":"2024-03-21T08:44:12.995621+0000","last_fullsized":"2024-03-21T08:44:12.995621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1253376,"data_stored":884902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":638976,"data_stored":287901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":659456,"data_stored":294430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":295026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":304199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":287957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1261568,"data_stored":895149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:44:19.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-21T08:44:19.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 Processor -- start 2024-03-21T08:44:19.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 -- start start 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 0x7fa5b40cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40d0380 con 0x7fa5b40b0ae0 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40d0660 con 0x7fa5b409a140 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5baa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d0150 con 0x7fa5b406c490 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5b9600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5b40b0ae0 0x7fa5b4099a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40006/0 (socket says 172.21.15.28:40006) 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5b9600640 1 -- 172.21.15.28:0/1012341308 learned_addr learned my addr 172.21.15.28:0/1012341308 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5ba000640 1 --2- 172.21.15.28:0/1012341308 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 0x7fa5b40cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149564286 0 0) 0x7fa5b40d0380 con 0x7fa5b40b0ae0 2024-03-21T08:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c003530 con 0x7fa5b40b0ae0 2024-03-21T08:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5ba000640 1 -- 172.21.15.28:0/1012341308 >> v1:172.21.15.151:6789/0 conn(0x7fa5b409a140 legacy=0x7fa5b415d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196777496 0 0) 0x7fa5b40d0660 con 0x7fa5b409a140 2024-03-21T08:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5ba000640 1 -- 172.21.15.28:0/1012341308 >> v1:172.21.15.28:6789/0 conn(0x7fa5b40b0ae0 legacy=0x7fa5b4099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.568+0000 7fa5ba000640 1 -- 172.21.15.28:0/1012341308 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40d0940 con 0x7fa5b406c490 2024-03-21T08:44:19.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5ba000640 1 --2- 172.21.15.28:0/1012341308 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 0x7fa5b40cfc10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009dc0 tx=0x7fa5ac01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d03f3e8a8f57e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0b2070 con 0x7fa5b406c490 2024-03-21T08:44:19.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac0a5850 con 0x7fa5b406c490 2024-03-21T08:44:19.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5b3e00640 1 -- 172.21.15.28:0/1012341308 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0a5c20 con 0x7fa5b406c490 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 -- 172.21.15.28:0/1012341308 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 msgr2=0x7fa5b40cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 --2- 172.21.15.28:0/1012341308 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 0x7fa5b40cfc10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009dc0 tx=0x7fa5ac01c990 comp rx=0 tx=0).stop 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 -- 172.21.15.28:0/1012341308 shutdown_connections 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 --2- 172.21.15.28:0/1012341308 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b406c490 0x7fa5b40cfc10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 -- 172.21.15.28:0/1012341308 >> 172.21.15.28:0/1012341308 conn(0x7fa5b40b3390 msgr2=0x7fa5b40b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 -- 172.21.15.28:0/1012341308 shutdown_connections 2024-03-21T08:44:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.572+0000 7fa5baa00640 1 -- 172.21.15.28:0/1012341308 wait complete. 2024-03-21T08:44:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 Processor -- start 2024-03-21T08:44:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- start start 2024-03-21T08:44:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b409a140 0x7fa5b415e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b40b0ae0 0x7fa5b415e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40cd080 con 0x7fa5b406c490 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40ccf00 con 0x7fa5b409a140 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415e030 con 0x7fa5b40b0ae0 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44002/0 (socket says 172.21.15.28:44002) 2024-03-21T08:44:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 -- 172.21.15.28:0/2266471087 learned_addr learned my addr 172.21.15.28:0/2266471087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 -- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b40b0ae0 msgr2=0x7fa5b415e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b40b0ae0 0x7fa5b415e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 -- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b409a140 msgr2=0x7fa5b415e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b409a140 0x7fa5b415e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 -- 172.21.15.28:0/2266471087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b41518f0 con 0x7fa5b406c490 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b9600640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800c3c0 tx=0x7fa5a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8017070 con 0x7fa5b406c490 2024-03-21T08:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8005db0 con 0x7fa5b406c490 2024-03-21T08:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b4151bd0 con 0x7fa5b406c490 2024-03-21T08:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80072f0 con 0x7fa5b406c490 2024-03-21T08:44:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b4152190 con 0x7fa5b406c490 2024-03-21T08:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.576+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4075270 con 0x7fa5b406c490 2024-03-21T08:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.580+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa5a801a070 con 0x7fa5b406c490 2024-03-21T08:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.580+0000 7fa5b1600640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 0x7fa594069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.580+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa5a8085160 con 0x7fa5b406c490 2024-03-21T08:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.580+0000 7fa5b8c00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 0x7fa594069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.580+0000 7fa5b8c00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 0x7fa594069fe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0007fd0 tx=0x7fa5a0007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:19.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.584+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa5a8055d70 con 0x7fa5b406c490 2024-03-21T08:44:19.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.748+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b4073f60 con 0x7fa5b406c490 2024-03-21T08:44:19.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.748+0000 7fa5b1600640 1 -- 172.21.15.28:0/2266471087 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13303 (secure 0 0 0) 0x7fa5a8055670 con 0x7fa5b406c490 2024-03-21T08:44:19.749 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:19.749 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":57,"fsid":"51dba44a-cba1-4b57-a836-26a407968834","created":"2024-03-21T08:40:42.615260+0000","modified":"2024-03-21T08:44:16.204496+0000","last_up_change":"2024-03-21T08:40:53.889834+0000","last_in_change":"2024-03-21T08:40:45.116957+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T08:40:53.789978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b33d81c-4e5b-47ee-8d63-94fc0d309fdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6809","nonce":1471116164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6811","nonce":1471116164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6818","nonce":1471116164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1471116164},{"type":"v1","addr":"172.21.15.28:6813","nonce":1471116164}]},"public_addr":"172.21.15.28:6809/1471116164","cluster_addr":"172.21.15.28:6811/1471116164","heartbeat_back_addr":"172.21.15.28:6818/1471116164","heartbeat_front_addr":"172.21.15.28:6813/1471116164","state":["exists","up"]},{"osd":1,"uuid":"e302143c-aff4-4d4a-8816-61a78f96bd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6801","nonce":4044131930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6803","nonce":4044131930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6807","nonce":4044131930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4044131930},{"type":"v1","addr":"172.21.15.28:6805","nonce":4044131930}]},"public_addr":"172.21.15.28:6801/4044131930","cluster_addr":"172.21.15.28:6803/4044131930","heartbeat_back_addr":"172.21.15.28:6807/4044131930","heartbeat_front_addr":"172.21.15.28:6805/4044131930","state":["exists","up"]},{"osd":2,"uuid":"a3d1a3f3-3ddb-482a-8498-45e23ce8f7bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6819","nonce":2368333363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6823","nonce":2368333363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6829","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6831","nonce":2368333363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":2368333363},{"type":"v1","addr":"172.21.15.28:6827","nonce":2368333363}]},"public_addr":"172.21.15.28:6819/2368333363","cluster_addr":"172.21.15.28:6823/2368333363","heartbeat_back_addr":"172.21.15.28:6831/2368333363","heartbeat_front_addr":"172.21.15.28:6827/2368333363","state":["exists","up"]},{"osd":3,"uuid":"cc23f2f0-5e4e-48a8-a39b-84c0451e4630","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6817","nonce":2425819575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6822","nonce":2425819575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6830","nonce":2425819575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2425819575},{"type":"v1","addr":"172.21.15.28:6826","nonce":2425819575}]},"public_addr":"172.21.15.28:6817/2425819575","cluster_addr":"172.21.15.28:6822/2425819575","heartbeat_back_addr":"172.21.15.28:6830/2425819575","heartbeat_front_addr":"172.21.15.28:6826/2425819575","state":["exists","up"]},{"osd":4,"uuid":"8f556bd4-249e-46ed-9ce0-6ac5e31994e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6809","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6811","nonce":3622853820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6813","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6815","nonce":3622853820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6821","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6823","nonce":3622853820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6817","nonce":3622853820},{"type":"v1","addr":"172.21.15.151:6819","nonce":3622853820}]},"public_addr":"172.21.15.151:6811/3622853820","cluster_addr":"172.21.15.151:6815/3622853820","heartbeat_back_addr":"172.21.15.151:6823/3622853820","heartbeat_front_addr":"172.21.15.151:6819/3622853820","state":["exists","up"]},{"osd":5,"uuid":"6d587fed-3a5a-402c-b87e-d16a3c1d36aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6801","nonce":2309128518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6803","nonce":2309128518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6807","nonce":2309128518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2309128518},{"type":"v1","addr":"172.21.15.151:6805","nonce":2309128518}]},"public_addr":"172.21.15.151:6801/2309128518","cluster_addr":"172.21.15.151:6803/2309128518","heartbeat_back_addr":"172.21.15.151:6807/2309128518","heartbeat_front_addr":"172.21.15.151:6805/2309128518","state":["exists","up"]},{"osd":6,"uuid":"a89375db-4a99-4718-b176-616941009b89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6810","nonce":3306491257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6814","nonce":3306491257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6822","nonce":3306491257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":3306491257},{"type":"v1","addr":"172.21.15.151:6818","nonce":3306491257}]},"public_addr":"172.21.15.151:6810/3306491257","cluster_addr":"172.21.15.151:6814/3306491257","heartbeat_back_addr":"172.21.15.151:6822/3306491257","heartbeat_front_addr":"172.21.15.151:6818/3306491257","state":["exists","up"]},{"osd":7,"uuid":"35d3b7a1-ce18-4615-8c1e-4f9ef04e5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6825","nonce":1764908544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6827","nonce":1764908544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6831","nonce":1764908544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1764908544},{"type":"v1","addr":"172.21.15.151:6829","nonce":1764908544}]},"public_addr":"172.21.15.151:6825/1764908544","cluster_addr":"172.21.15.151:6827/1764908544","heartbeat_back_addr":"172.21.15.151:6831/1764908544","heartbeat_front_addr":"172.21.15.151:6829/1764908544","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:51.877516+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.946157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.960090+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.761614+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.663401+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.646913+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.777201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T08:40:50.886289+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/460337969":"2024-03-22T08:44:12.990134+0000","172.21.15.28:6834/460337969":"2024-03-22T08:44:12.990134+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T08:44:19.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 msgr2=0x7fa594069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 0x7fa594069fe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0007fd0 tx=0x7fa5a0007920 comp rx=0 tx=0).stop 2024-03-21T08:44:19.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 msgr2=0x7fa5b41400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:19.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800c3c0 tx=0x7fa5a800c880 comp rx=0 tx=0).stop 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 shutdown_connections 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa594067b20 0x7fa594069fe0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa5b40b0ae0 0x7fa5b415e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa5b409a140 0x7fa5b415e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 --2- 172.21.15.28:0/2266471087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b406c490 0x7fa5b41400f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 >> 172.21.15.28:0/2266471087 conn(0x7fa5b40b3390 msgr2=0x7fa5b4099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 shutdown_connections 2024-03-21T08:44:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:19.752+0000 7fa5baa00640 1 -- 172.21.15.28:0/2266471087 wait complete. 2024-03-21T08:44:20.771 INFO:tasks.ceph:Scrubbing osd.0 2024-03-21T08:44:20.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:20.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 Processor -- start 2024-03-21T08:44:20.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 -- start start 2024-03-21T08:44:20.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 0x7f0da406bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da4067990 con 0x7f0da4099a30 2024-03-21T08:44:20.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da415e370 con 0x7f0da415df90 2024-03-21T08:44:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da9e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da406b680 con 0x7f0da406ea90 2024-03-21T08:44:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0da4099a30 0x7f0da406c050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50448/0 (socket says 172.21.15.28:50448) 2024-03-21T08:44:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da3400640 1 -- 172.21.15.28:0/4111696009 learned_addr learned my addr 172.21.15.28:0/4111696009 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0d9be00640 1 --2- 172.21.15.28:0/4111696009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 0x7f0da406bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890914466 0 0) 0x7f0d9c0014a0 con 0x7f0da4099a30 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d88003530 con 0x7f0da4099a30 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333447846 0 0) 0x7f0da415e370 con 0x7f0da415df90 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0d9be00640 1 -- 172.21.15.28:0/4111696009 >> v1:172.21.15.151:6789/0 conn(0x7f0da415df90 legacy=0x7f0da406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0d9be00640 1 -- 172.21.15.28:0/4111696009 >> v1:172.21.15.28:6789/0 conn(0x7f0da4099a30 legacy=0x7f0da406c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.928+0000 7f0d9be00640 1 -- 172.21.15.28:0/4111696009 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da40af700 con 0x7f0da406ea90 2024-03-21T08:44:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0d9be00640 1 --2- 172.21.15.28:0/4111696009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 0x7f0da406bb10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009820 tx=0x7f0d8c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafe2e2d3f1dc0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0b0070 con 0x7f0da406ea90 2024-03-21T08:44:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8c0ab600 con 0x7f0da406ea90 2024-03-21T08:44:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da2000640 1 -- 172.21.15.28:0/4111696009 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0ab9b0 con 0x7f0da406ea90 2024-03-21T08:44:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 -- 172.21.15.28:0/4111696009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 msgr2=0x7f0da406bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 --2- 172.21.15.28:0/4111696009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 0x7f0da406bb10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009820 tx=0x7f0d8c01c4c0 comp rx=0 tx=0).stop 2024-03-21T08:44:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 -- 172.21.15.28:0/4111696009 shutdown_connections 2024-03-21T08:44:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 --2- 172.21.15.28:0/4111696009 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da406ea90 0x7f0da406bb10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 -- 172.21.15.28:0/4111696009 >> 172.21.15.28:0/4111696009 conn(0x7f0da406daf0 msgr2=0x7f0da40b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 -- 172.21.15.28:0/4111696009 shutdown_connections 2024-03-21T08:44:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.932+0000 7f0da9e00640 1 -- 172.21.15.28:0/4111696009 wait complete. 2024-03-21T08:44:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 Processor -- start 2024-03-21T08:44:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- start start 2024-03-21T08:44:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da406ea90 0x7f0da4143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da4099a30 0x7f0da4142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 0x7f0da4147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40b3d50 con 0x7f0da415df90 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40b3bd0 con 0x7f0da406ea90 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da416c620 con 0x7f0da4099a30 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 0x7f0da4147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 0x7f0da4147ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33214/0 (socket says 172.21.15.28:33214) 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 -- 172.21.15.28:0/3570197417 learned_addr learned my addr 172.21.15.28:0/3570197417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da406ea90 0x7f0da4143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0d9be00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da4099a30 0x7f0da4142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 -- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da4099a30 msgr2=0x7f0da4142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f0da4099a30 0x7f0da4142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 -- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da406ea90 msgr2=0x7f0da4143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f0da406ea90 0x7f0da4143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da41488e0 con 0x7f0da415df90 2024-03-21T08:44:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da3400640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 0x7f0da4147ee0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00cb40 tx=0x7f0d9c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c017070 con 0x7f0da415df90 2024-03-21T08:44:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c00ac40 con 0x7f0da415df90 2024-03-21T08:44:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c012400 con 0x7f0da415df90 2024-03-21T08:44:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da4148bc0 con 0x7f0da415df90 2024-03-21T08:44:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.936+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da4144510 con 0x7f0da415df90 2024-03-21T08:44:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0da4143bb0 con 0x7f0da415df90 2024-03-21T08:44:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f0d9c01a070 con 0x7f0da415df90 2024-03-21T08:44:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0d80067b20 0x7f0d80069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0d9c085ad0 con 0x7f0da415df90 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f0d8006dfc0 0x7f0d80070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d80070ad0 con 0x7f0d8006dfc0 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d9c08a2a0 con 0x7f0da415df90 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0d80067b20 0x7f0d80069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9be00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f0d8006dfc0 0x7f0d80070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0d80067b20 0x7f0d80069fe0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0d94007fd0 tx=0x7f0d94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.940+0000 7f0d9be00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f0d8006dfc0 0x7f0d80070400 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.944+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== osd.0 v2:172.21.15.28:6808/1471116164 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d80070ad0 con 0x7f0d8006dfc0 2024-03-21T08:44:20.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 --> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0da4068110 con 0x7f0d8006dfc0 2024-03-21T08:44:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0d9b400640 1 -- 172.21.15.28:0/3570197417 <== osd.0 v2:172.21.15.28:6808/1471116164 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0da4068110 con 0x7f0d8006dfc0 2024-03-21T08:44:20.957 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:20.957 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f0d8006dfc0 msgr2=0x7f0d80070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:6808/1471116164,v1:172.21.15.28:6809/1471116164] conn(0x7f0d8006dfc0 0x7f0d80070400 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0d80067b20 msgr2=0x7f0d80069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0d80067b20 0x7f0d80069fe0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0d94007fd0 tx=0x7f0d94009040 comp rx=0 tx=0).stop 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 msgr2=0x7f0da4147ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 --2- 172.21.15.28:0/3570197417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da415df90 0x7f0da4147ee0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00cb40 tx=0x7f0d9c00a060 comp rx=0 tx=0).stop 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da3400640 1 -- 172.21.15.28:0/3570197417 reap_dead start 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 shutdown_connections 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 >> 172.21.15.28:0/3570197417 conn(0x7f0da406daf0 msgr2=0x7f0da4155c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 shutdown_connections 2024-03-21T08:44:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:20.956+0000 7f0da9e00640 1 -- 172.21.15.28:0/3570197417 wait complete. 2024-03-21T08:44:20.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-21T08:44:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.116+0000 7fa352600640 1 Processor -- start 2024-03-21T08:44:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.116+0000 7fa352600640 1 -- start start 2024-03-21T08:44:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa352600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0320 0x7fa34c0d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa352600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa352600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0d0c40 con 0x7fa34c15e190 2024-03-21T08:44:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa352600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c06bd40 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa352600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c06c020 con 0x7fa34c0d0320 2024-03-21T08:44:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33218/0 (socket says 172.21.15.28:33218) 2024-03-21T08:44:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 -- 172.21.15.28:0/3720256262 learned_addr learned my addr 172.21.15.28:0/3720256262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34b400640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0320 0x7fa34c0d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 -- 172.21.15.28:0/3720256262 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0320 msgr2=0x7fa34c0d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0320 0x7fa34c0d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 -- 172.21.15.28:0/3720256262 >> v1:172.21.15.151:6789/0 conn(0x7fa34c15e190 legacy=0x7fa34c0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.120+0000 7fa34be00640 1 -- 172.21.15.28:0/3720256262 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c06c300 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa34be00640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa34401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838e03e5b30de353 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa34aa00640 1 -- 172.21.15.28:0/3720256262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440a5770 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa34aa00640 1 -- 172.21.15.28:0/3720256262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440a5910 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa34aa00640 1 -- 172.21.15.28:0/3720256262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440a5ce0 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- 172.21.15.28:0/3720256262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c0d0ee0 msgr2=0x7fa34c0ad2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa34401c9a0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- 172.21.15.28:0/3720256262 shutdown_connections 2024-03-21T08:44:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c0d0ee0 0x7fa34c0ad2b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- 172.21.15.28:0/3720256262 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0320 0x7fa34c0d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- 172.21.15.28:0/3720256262 >> 172.21.15.28:0/3720256262 conn(0x7fa34c099a30 msgr2=0x7fa34c0b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- 172.21.15.28:0/3720256262 shutdown_connections 2024-03-21T08:44:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- 172.21.15.28:0/3720256262 wait complete. 2024-03-21T08:44:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 Processor -- start 2024-03-21T08:44:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 -- start start 2024-03-21T08:44:21.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa34c0d0320 0x7fa34c13c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 0x7fa34c155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.124+0000 7fa352600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0ae390 con 0x7fa34c15e190 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0ae210 con 0x7fa34c0d0320 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0ae510 con 0x7fa34c0d0ee0 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33228/0 (socket says 172.21.15.28:33228) 2024-03-21T08:44:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 -- 172.21.15.28:0/445307995 learned_addr learned my addr 172.21.15.28:0/445307995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34b400640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa34c0d0320 0x7fa34c13c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343e00640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 0x7fa34c155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 -- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 msgr2=0x7fa34c155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 0x7fa34c155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 -- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa34c0d0320 msgr2=0x7fa34c13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa34c0d0320 0x7fa34c13c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 -- 172.21.15.28:0/445307995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c13f560 con 0x7fa34c15e190 2024-03-21T08:44:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343e00640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 0x7fa34c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa34be00640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa34401a5d0 tx=0x7fa344019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440bc020 con 0x7fa34c15e190 2024-03-21T08:44:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440b8880 con 0x7fa34c15e190 2024-03-21T08:44:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440b8c70 con 0x7fa34c15e190 2024-03-21T08:44:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c13f840 con 0x7fa34c15e190 2024-03-21T08:44:21.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c13fe00 con 0x7fa34c15e190 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c04f4f0 con 0x7fa34c15e190 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa3440bf020 con 0x7fa34c15e190 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.128+0000 7fa343400640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 0x7fa328069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.132+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa344120fd0 con 0x7fa34c15e190 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.132+0000 7fa34b400640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 0x7fa328069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.132+0000 7fa34b400640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 0x7fa328069fe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa33c006aa0 tx=0x7fa33c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.132+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa3440c6030 con 0x7fa34c15e190 2024-03-21T08:44:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.296+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa34c0d0100 con 0x7fa328067b20 2024-03-21T08:44:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.296+0000 7fa343400640 1 -- 172.21.15.28:0/445307995 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa34c0d0100 con 0x7fa328067b20 2024-03-21T08:44:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 msgr2=0x7fa328069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 0x7fa328069fe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa33c006aa0 tx=0x7fa33c008040 comp rx=0 tx=0).stop 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 msgr2=0x7fa34c0af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa34401a5d0 tx=0x7fa344019740 comp rx=0 tx=0).stop 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 shutdown_connections 2024-03-21T08:44:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34c15e190 0x7fa34c0af480 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa328067b20 0x7fa328069fe0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa34c0d0ee0 0x7fa34c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 --2- 172.21.15.28:0/445307995 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa34c0d0320 0x7fa34c13c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 >> 172.21.15.28:0/445307995 conn(0x7fa34c099a30 msgr2=0x7fa34c0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 shutdown_connections 2024-03-21T08:44:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.300+0000 7fa352600640 1 -- 172.21.15.28:0/445307995 wait complete. 2024-03-21T08:44:21.319 INFO:tasks.ceph:Scrubbing osd.1 2024-03-21T08:44:21.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:21.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 Processor -- start 2024-03-21T08:44:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 -- start start 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94c40ce840 0x7f94c40cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4161a80 con 0x7f94c40cfa00 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c40cfe70 con 0x7f94c40ce840 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c3e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c406cd50 con 0x7f94c415daf0 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94c40ce840 0x7f94c40cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94c40ce840 0x7f94c40cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33242/0 (socket says 172.21.15.28:33242) 2024-03-21T08:44:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94ba000640 1 -- 172.21.15.28:0/3616440291 learned_addr learned my addr 172.21.15.28:0/3616440291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.464+0000 7f94c2a00640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94ba000640 1 -- 172.21.15.28:0/3616440291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 msgr2=0x7f94c40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94ba000640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c40ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94ba000640 1 -- 172.21.15.28:0/3616440291 >> v1:172.21.15.151:6789/0 conn(0x7f94c40cfa00 legacy=0x7f94c4071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94ba000640 1 -- 172.21.15.28:0/3616440291 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c406c740 con 0x7f94c40ce840 2024-03-21T08:44:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94ba000640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c40cf4c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d208401e6f358c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c2000640 1 -- 172.21.15.28:0/3616440291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0b5070 con 0x7f94c40ce840 2024-03-21T08:44:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c2000640 1 -- 172.21.15.28:0/3616440291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac0a8a50 con 0x7f94c40ce840 2024-03-21T08:44:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c2000640 1 -- 172.21.15.28:0/3616440291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0a8e20 con 0x7f94c40ce840 2024-03-21T08:44:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 -- 172.21.15.28:0/3616440291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 msgr2=0x7f94c40cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c40cf4c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01ed50 comp rx=0 tx=0).stop 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 -- 172.21.15.28:0/3616440291 shutdown_connections 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c40cf4c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 --2- 172.21.15.28:0/3616440291 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c40ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 -- 172.21.15.28:0/3616440291 >> 172.21.15.28:0/3616440291 conn(0x7f94c406fa90 msgr2=0x7f94c406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 -- 172.21.15.28:0/3616440291 shutdown_connections 2024-03-21T08:44:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.468+0000 7f94c3e00640 1 -- 172.21.15.28:0/3616440291 wait complete. 2024-03-21T08:44:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 Processor -- start 2024-03-21T08:44:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- start start 2024-03-21T08:44:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94c40cfa00 0x7f94c40cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c415fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40d0380 con 0x7f94c40ce840 2024-03-21T08:44:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40cec80 con 0x7f94c40cfa00 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c406c9e0 con 0x7f94c415daf0 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c415ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33244/0 (socket says 172.21.15.28:33244) 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 -- 172.21.15.28:0/2176128423 learned_addr learned my addr 172.21.15.28:0/2176128423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94ba000640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94c40cfa00 0x7f94c40cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c415fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 -- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 msgr2=0x7f94c415fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c415fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 -- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94c40cfa00 msgr2=0x7f94c40cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94c40cfa00 0x7f94c40cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c415ecc0 con 0x7f94c40ce840 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94ba000640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f94c40cfa00 0x7f94c40cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f94c415daf0 0x7f94c415fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c2a00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c415ae10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f94b400e870 tx=0x7f94b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4017070 con 0x7f94c40ce840 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b400c940 con 0x7f94c40ce840 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b400ccf0 con 0x7f94c40ce840 2024-03-21T08:44:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c415efa0 con 0x7f94c40ce840 2024-03-21T08:44:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.472+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c415f560 con 0x7f94c40ce840 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94c404f410 con 0x7f94c40ce840 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f94b401a070 con 0x7f94c40ce840 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f94a0067b20 0x7f94a0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f94b40863b0 con 0x7f94c40ce840 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f94a006dfc0 0x7f94a0070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94b4013e50 con 0x7f94a006dfc0 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94b4086840 con 0x7f94c40ce840 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94ba000640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f94a0067b20 0x7f94a0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94c3400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f94a006dfc0 0x7f94a0070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94c3400640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f94a006dfc0 0x7f94a0070400 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.476+0000 7f94ba000640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f94a0067b20 0x7f94a0069fe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0b2000 tx=0x7f94ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.480+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== osd.1 v2:172.21.15.28:6800/4044131930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f94b4013e50 con 0x7f94a006dfc0 2024-03-21T08:44:21.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 --> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f94c40c89c0 con 0x7f94a006dfc0 2024-03-21T08:44:21.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94bb400640 1 -- 172.21.15.28:0/2176128423 <== osd.1 v2:172.21.15.28:6800/4044131930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f94c40c89c0 con 0x7f94a006dfc0 2024-03-21T08:44:21.493 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:21.493 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:21.494 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:21.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f94a006dfc0 msgr2=0x7f94a0070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:6800/4044131930,v1:172.21.15.28:6801/4044131930] conn(0x7f94a006dfc0 0x7f94a0070400 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f94a0067b20 msgr2=0x7f94a0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f94a0067b20 0x7f94a0069fe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0b2000 tx=0x7f94ac008040 comp rx=0 tx=0).stop 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 msgr2=0x7f94c415ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 --2- 172.21.15.28:0/2176128423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94c40ce840 0x7f94c415ae10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f94b400e870 tx=0x7f94b400ec70 comp rx=0 tx=0).stop 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3400640 1 -- 172.21.15.28:0/2176128423 reap_dead start 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 shutdown_connections 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 >> 172.21.15.28:0/2176128423 conn(0x7f94c406fa90 msgr2=0x7f94c4161c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 shutdown_connections 2024-03-21T08:44:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.492+0000 7f94c3e00640 1 -- 172.21.15.28:0/2176128423 wait complete. 2024-03-21T08:44:21.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-21T08:44:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.652+0000 7fd33be00640 1 Processor -- start 2024-03-21T08:44:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.652+0000 7fd33be00640 1 -- start start 2024-03-21T08:44:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.652+0000 7fd33be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd33406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd33be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3340ce5b0 con 0x7fd3340cde40 2024-03-21T08:44:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd33be00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3340ce890 con 0x7fd33415cbb0 2024-03-21T08:44:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd33be00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33415c720 con 0x7fd334071a80 2024-03-21T08:44:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd338a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd3340cde40 0x7fd334071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50460/0 (socket says 172.21.15.28:50460) 2024-03-21T08:44:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd338a00640 1 -- 172.21.15.28:0/1974875521 learned_addr learned my addr 172.21.15.28:0/1974875521 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd33406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770604320 0 0) 0x7fd3340ce5b0 con 0x7fd3340cde40 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd31c003530 con 0x7fd3340cde40 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2926304947 0 0) 0x7fd31c003530 con 0x7fd3340cde40 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 msgr2=0x7fd33406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:21.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd33406f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 >> v1:172.21.15.151:6789/0 conn(0x7fd33415cbb0 legacy=0x7fd3340cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33415c3b0 con 0x7fd3340cde40 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fd320001680 con 0x7fd3340cde40 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 >> v1:172.21.15.28:6789/0 conn(0x7fd3340cde40 legacy=0x7fd334071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c0039f0 0x7fd31c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0043a0 0x7fd31c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd31c005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33415c720 con 0x7fd31c0039f0 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c00ba80 con 0x7fd31c0043a0 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c00bc80 con 0x7fd334071a80 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c0039f0 0x7fd31c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd338a00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0043a0 0x7fd31c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339e00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd31c005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 -- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 msgr2=0x7fd31c005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd334071a80 0x7fd31c005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 -- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0043a0 msgr2=0x7fd31c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0043a0 0x7fd31c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.656+0000 7fd339400640 1 -- 172.21.15.28:0/1974875521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c003530 con 0x7fd31c0039f0 2024-03-21T08:44:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd339400640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c0039f0 0x7fd31c003dd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd328009ae0 tx=0x7fd32801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726387a77f31aa7f server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3280b2070 con 0x7fd31c0039f0 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3280a5670 con 0x7fd31c0039f0 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd333e00640 1 -- 172.21.15.28:0/1974875521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3280096b0 con 0x7fd31c0039f0 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- 172.21.15.28:0/1974875521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c0039f0 msgr2=0x7fd31c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 --2- 172.21.15.28:0/1974875521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c0039f0 0x7fd31c003dd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd328009ae0 tx=0x7fd32801ca20 comp rx=0 tx=0).stop 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd339e00640 1 -- 172.21.15.28:0/1974875521 reap_dead start 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- 172.21.15.28:0/1974875521 shutdown_connections 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- 172.21.15.28:0/1974875521 >> 172.21.15.28:0/1974875521 conn(0x7fd3340674f0 msgr2=0x7fd33406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- 172.21.15.28:0/1974875521 shutdown_connections 2024-03-21T08:44:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- 172.21.15.28:0/1974875521 wait complete. 2024-03-21T08:44:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 Processor -- start 2024-03-21T08:44:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.660+0000 7fd33be00640 1 -- start start 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0039f0 0x7fd3341f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3340b05b0 0x7fd3340b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406f220 con 0x7fd3341e74f0 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406f0a0 con 0x7fd31c0039f0 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406f3a0 con 0x7fd3340b05b0 2024-03-21T08:44:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33270/0 (socket says 172.21.15.28:33270) 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0039f0 0x7fd3341f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 -- 172.21.15.28:0/1732283094 learned_addr learned my addr 172.21.15.28:0/1732283094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd338a00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3340b05b0 0x7fd3340b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 -- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3340b05b0 msgr2=0x7fd3340b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3340b05b0 0x7fd3340b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 -- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0039f0 msgr2=0x7fd3341f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0039f0 0x7fd3341f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 -- 172.21.15.28:0/1732283094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3341e82b0 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd339e00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd33406c130 tx=0x7fd32c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c006b60 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c006d00 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c003bc0 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33415f560 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33415fb20 con 0x7fd3341e74f0 2024-03-21T08:44:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33404f4f0 con 0x7fd3341e74f0 2024-03-21T08:44:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.664+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fd32c005110 con 0x7fd3341e74f0 2024-03-21T08:44:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.668+0000 7fd331600640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 0x7fd314069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.668+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd32c085530 con 0x7fd3341e74f0 2024-03-21T08:44:21.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.668+0000 7fd339400640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 0x7fd314069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:21.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.668+0000 7fd339400640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 0x7fd314069fe0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd3280af000 tx=0x7fd3280046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.672+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd32c055a60 con 0x7fd3341e74f0 2024-03-21T08:44:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.832+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd334073410 con 0x7fd314067b20 2024-03-21T08:44:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.832+0000 7fd331600640 1 -- 172.21.15.28:0/1732283094 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd334073410 con 0x7fd314067b20 2024-03-21T08:44:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-03-21T08:44:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 msgr2=0x7fd314069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 0x7fd314069fe0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd3280af000 tx=0x7fd3280046d0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 msgr2=0x7fd3341e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd33406c130 tx=0x7fd32c00eb90 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 shutdown_connections 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3341e74f0 0x7fd3341e7970 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fd314067b20 0x7fd314069fe0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fd3340b05b0 0x7fd3340b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 --2- 172.21.15.28:0/1732283094 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd31c0039f0 0x7fd3341f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 >> 172.21.15.28:0/1732283094 conn(0x7fd3340674f0 msgr2=0x7fd3340b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 shutdown_connections 2024-03-21T08:44:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:21.836+0000 7fd33be00640 1 -- 172.21.15.28:0/1732283094 wait complete. 2024-03-21T08:44:21.855 INFO:tasks.ceph:Scrubbing osd.2 2024-03-21T08:44:21.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 Processor -- start 2024-03-21T08:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 -- start start 2024-03-21T08:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f92d8072130 0x7f92d815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d806da60 con 0x7f92d815c150 2024-03-21T08:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d815cc80 con 0x7f92d80ce6d0 2024-03-21T08:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92e0200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d815cf60 con 0x7f92d8072130 2024-03-21T08:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92dce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f92d815c150 0x7f92d8071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50468/0 (socket says 172.21.15.28:50468) 2024-03-21T08:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92dce00640 1 -- 172.21.15.28:0/3177202340 learned_addr learned my addr 172.21.15.28:0/3177202340 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92de200640 1 --2- 172.21.15.28:0/3177202340 >> v2:172.21.15.151:3300/0 conn(0x7f92d8072130 0x7f92d815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705850671 0 0) 0x7f92d806da60 con 0x7f92d815c150 2024-03-21T08:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c0003530 con 0x7f92d815c150 2024-03-21T08:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92de200640 1 -- 172.21.15.28:0/3177202340 >> v1:172.21.15.151:6789/0 conn(0x7f92d80ce6d0 legacy=0x7f92d806e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354685067 0 0) 0x7f92d815cc80 con 0x7f92d80ce6d0 2024-03-21T08:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92de200640 1 -- 172.21.15.28:0/3177202340 >> v1:172.21.15.28:6789/0 conn(0x7f92d815c150 legacy=0x7f92d8071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.000+0000 7f92de200640 1 -- 172.21.15.28:0/3177202340 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d806dd20 con 0x7f92d8072130 2024-03-21T08:44:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92de200640 1 --2- 172.21.15.28:0/3177202340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d8072130 0x7f92d815c5b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009d90 tx=0x7f92d001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5ebba7c96955b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00b2070 con 0x7f92d8072130 2024-03-21T08:44:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d00a5820 con 0x7f92d8072130 2024-03-21T08:44:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92d7e00640 1 -- 172.21.15.28:0/3177202340 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00a5bf0 con 0x7f92d8072130 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 -- 172.21.15.28:0/3177202340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d8072130 msgr2=0x7f92d815c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 --2- 172.21.15.28:0/3177202340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d8072130 0x7f92d815c5b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009d90 tx=0x7f92d001c8e0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 -- 172.21.15.28:0/3177202340 shutdown_connections 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 --2- 172.21.15.28:0/3177202340 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d8072130 0x7f92d815c5b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 -- 172.21.15.28:0/3177202340 >> 172.21.15.28:0/3177202340 conn(0x7f92d80674f0 msgr2=0x7f92d806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 -- 172.21.15.28:0/3177202340 shutdown_connections 2024-03-21T08:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.004+0000 7f92e0200640 1 -- 172.21.15.28:0/3177202340 wait complete. 2024-03-21T08:44:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 Processor -- start 2024-03-21T08:44:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 -- start start 2024-03-21T08:44:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92d8072130 0x7f92d80b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d80ce6d0 0x7f92d8152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 0x7f92d815e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8169af0 con 0x7f92d815c150 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80ccea0 con 0x7f92d80ce6d0 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8169c70 con 0x7f92d8072130 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 0x7f92d815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 0x7f92d815e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33274/0 (socket says 172.21.15.28:33274) 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 -- 172.21.15.28:0/2584660829 learned_addr learned my addr 172.21.15.28:0/2584660829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92dce00640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d80ce6d0 0x7f92d8152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92dd800640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92d8072130 0x7f92d80b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 -- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92d8072130 msgr2=0x7f92d80b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f92d8072130 0x7f92d80b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 -- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d80ce6d0 msgr2=0x7f92d8152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f92d80ce6d0 0x7f92d8152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d8072980 con 0x7f92d815c150 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92de200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 0x7f92d815e980 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f92d00b0040 tx=0x7f92d0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00c0070 con 0x7f92d815c150 2024-03-21T08:44:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d00039a0 con 0x7f92d815c150 2024-03-21T08:44:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d00b2040 con 0x7f92d815c150 2024-03-21T08:44:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.008+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d8139d00 con 0x7f92d815c150 2024-03-21T08:44:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d81527f0 con 0x7f92d815c150 2024-03-21T08:44:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92d8152bd0 con 0x7f92d815c150 2024-03-21T08:44:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f92d0004030 con 0x7f92d815c150 2024-03-21T08:44:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f92b8067b50 0x7f92b806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f92d0120780 con 0x7f92d815c150 2024-03-21T08:44:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f92b806dff0 0x7f92b8070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92d00c9ea0 con 0x7f92b806dff0 2024-03-21T08:44:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92d0120b70 con 0x7f92d815c150 2024-03-21T08:44:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92dd800640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f92b8067b50 0x7f92b806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92dce00640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f92b806dff0 0x7f92b8070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92dd800640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f92b8067b50 0x7f92b806a010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000f70 tx=0x7f92cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.012+0000 7f92dce00640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f92b806dff0 0x7f92b8070430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.016+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== osd.2 v2:172.21.15.28:6815/2368333363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f92d00c9ea0 con 0x7f92b806dff0 2024-03-21T08:44:22.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 --> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f92d8068110 con 0x7f92b806dff0 2024-03-21T08:44:22.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92d5600640 1 -- 172.21.15.28:0/2584660829 <== osd.2 v2:172.21.15.28:6815/2368333363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f92d8068110 con 0x7f92b806dff0 2024-03-21T08:44:22.030 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:22.030 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:22.030 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:22.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f92b806dff0 msgr2=0x7f92b8070430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:6815/2368333363,v1:172.21.15.28:6819/2368333363] conn(0x7f92b806dff0 0x7f92b8070430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f92b8067b50 msgr2=0x7f92b806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f92b8067b50 0x7f92b806a010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000f70 tx=0x7f92cc009040 comp rx=0 tx=0).stop 2024-03-21T08:44:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 msgr2=0x7f92d815e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 --2- 172.21.15.28:0/2584660829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92d815c150 0x7f92d815e980 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f92d00b0040 tx=0x7f92d0019680 comp rx=0 tx=0).stop 2024-03-21T08:44:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92de200640 1 -- 172.21.15.28:0/2584660829 reap_dead start 2024-03-21T08:44:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 shutdown_connections 2024-03-21T08:44:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 >> 172.21.15.28:0/2584660829 conn(0x7f92d80674f0 msgr2=0x7f92d815e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 shutdown_connections 2024-03-21T08:44:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.028+0000 7f92e0200640 1 -- 172.21.15.28:0/2584660829 wait complete. 2024-03-21T08:44:22.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-21T08:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 Processor -- start 2024-03-21T08:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 -- start start 2024-03-21T08:44:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fbf3c072190 0x7fbf3c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c15bf90 con 0x7fbf3c0b4800 2024-03-21T08:44:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c15b1d0 con 0x7fbf3c0b4420 2024-03-21T08:44:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf41a00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c15b4b0 con 0x7fbf3c072190 2024-03-21T08:44:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf33e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbf3c0b4800 0x7fbf3c0ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50482/0 (socket says 172.21.15.28:50482) 2024-03-21T08:44:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf33e00640 1 -- 172.21.15.28:0/859527594 learned_addr learned my addr 172.21.15.28:0/859527594 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.188+0000 7fbf3b400640 1 --2- 172.21.15.28:0/859527594 >> v2:172.21.15.151:3300/0 conn(0x7fbf3c072190 0x7fbf3c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3a000640 1 -- 172.21.15.28:0/859527594 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866905866 0 0) 0x7fbf3c15bf90 con 0x7fbf3c0b4800 2024-03-21T08:44:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3a000640 1 -- 172.21.15.28:0/859527594 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf20003530 con 0x7fbf3c0b4800 2024-03-21T08:44:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3b400640 1 -- 172.21.15.28:0/859527594 >> v1:172.21.15.151:6789/0 conn(0x7fbf3c0b4420 legacy=0x7fbf3c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3b400640 1 -- 172.21.15.28:0/859527594 >> v1:172.21.15.28:6789/0 conn(0x7fbf3c0b4800 legacy=0x7fbf3c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3b400640 1 -- 172.21.15.28:0/859527594 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c15b790 con 0x7fbf3c072190 2024-03-21T08:44:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3b400640 1 --2- 172.21.15.28:0/859527594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c072190 0x7fbf3c0cd300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bc29a501b3a3816 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3a000640 1 -- 172.21.15.28:0/859527594 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340b2070 con 0x7fbf3c072190 2024-03-21T08:44:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3a000640 1 -- 172.21.15.28:0/859527594 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf340a59a0 con 0x7fbf3c072190 2024-03-21T08:44:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.192+0000 7fbf3a000640 1 -- 172.21.15.28:0/859527594 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340a5d70 con 0x7fbf3c072190 2024-03-21T08:44:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- 172.21.15.28:0/859527594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c072190 msgr2=0x7fbf3c0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 --2- 172.21.15.28:0/859527594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c072190 0x7fbf3c0cd300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3401c940 comp rx=0 tx=0).stop 2024-03-21T08:44:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- 172.21.15.28:0/859527594 shutdown_connections 2024-03-21T08:44:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 --2- 172.21.15.28:0/859527594 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c072190 0x7fbf3c0cd300 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- 172.21.15.28:0/859527594 >> 172.21.15.28:0/859527594 conn(0x7fbf3c0674f0 msgr2=0x7fbf3c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- 172.21.15.28:0/859527594 shutdown_connections 2024-03-21T08:44:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- 172.21.15.28:0/859527594 wait complete. 2024-03-21T08:44:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 Processor -- start 2024-03-21T08:44:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- start start 2024-03-21T08:44:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 0x7fbf3c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c164eb0 0x7fbf3c1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c168140 con 0x7fbf3c072190 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c167fc0 con 0x7fbf3c164eb0 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf41a00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c1682c0 con 0x7fbf3c0b4420 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33284/0 (socket says 172.21.15.28:33284) 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 -- 172.21.15.28:0/3864276164 learned_addr learned my addr 172.21.15.28:0/3864276164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3b400640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c164eb0 0x7fbf3c1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf33e00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 0x7fbf3c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 -- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 msgr2=0x7fbf3c14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 0x7fbf3c14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 -- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c164eb0 msgr2=0x7fbf3c1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c164eb0 0x7fbf3c1461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf3aa00640 1 -- 172.21.15.28:0/3864276164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c146a00 con 0x7fbf3c072190 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.196+0000 7fbf33e00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 0x7fbf3c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf3aa00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00c420 tx=0x7fbf2c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c0052d0 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c005470 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c14c480 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c007260 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c14ca40 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c073430 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fbf2c007400 con 0x7fbf3c072190 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 0x7fbf1806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fbf2c085210 con 0x7fbf3c072190 2024-03-21T08:44:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.200+0000 7fbf33e00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 0x7fbf1806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.204+0000 7fbf33e00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 0x7fbf1806a010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c14e7a0 tx=0x7fbf24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.204+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbf2c0556a0 con 0x7fbf3c072190 2024-03-21T08:44:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.368+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf3c1455d0 con 0x7fbf18067b50 2024-03-21T08:44:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf32a00640 1 -- 172.21.15.28:0/3864276164 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbf3c1455d0 con 0x7fbf18067b50 2024-03-21T08:44:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-03-21T08:44:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 msgr2=0x7fbf1806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 0x7fbf1806a010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c14e7a0 tx=0x7fbf24009040 comp rx=0 tx=0).stop 2024-03-21T08:44:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 msgr2=0x7fbf3c145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00c420 tx=0x7fbf2c00c8e0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 shutdown_connections 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fbf3c164eb0 0x7fbf3c1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fbf18067b50 0x7fbf1806a010 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fbf3c0b4420 0x7fbf3c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 --2- 172.21.15.28:0/3864276164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf3c072190 0x7fbf3c145090 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 >> 172.21.15.28:0/3864276164 conn(0x7fbf3c0674f0 msgr2=0x7fbf3c0cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 shutdown_connections 2024-03-21T08:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.372+0000 7fbf41a00640 1 -- 172.21.15.28:0/3864276164 wait complete. 2024-03-21T08:44:22.393 INFO:tasks.ceph:Scrubbing osd.3 2024-03-21T08:44:22.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 Processor -- start 2024-03-21T08:44:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 -- start start 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fcd000b0f60 0x7fcd000d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000d0040 con 0x7fcd0015ba60 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000d0320 con 0x7fcd000cf9b0 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd05800640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd000cfde0 con 0x7fcd000b0f60 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccffe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd0015ba60 0x7fcd000b0850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50490/0 (socket says 172.21.15.28:50490) 2024-03-21T08:44:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccffe00640 1 -- 172.21.15.28:0/268942047 learned_addr learned my addr 172.21.15.28:0/268942047 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 --2- 172.21.15.28:0/268942047 >> v2:172.21.15.151:3300/0 conn(0x7fcd000b0f60 0x7fcd000d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752191955 0 0) 0x7fcd000d0040 con 0x7fcd0015ba60 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce4003530 con 0x7fcd0015ba60 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421090817 0 0) 0x7fcce4003530 con 0x7fcd0015ba60 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 >> v2:172.21.15.151:3300/0 conn(0x7fcd000b0f60 msgr2=0x7fcd000d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 --2- 172.21.15.28:0/268942047 >> v2:172.21.15.151:3300/0 conn(0x7fcd000b0f60 0x7fcd000d10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 >> v1:172.21.15.151:6789/0 conn(0x7fcd000cf9b0 legacy=0x7fcd000b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000d0600 con 0x7fcd0015ba60 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7fccf0001680 con 0x7fcd0015ba60 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 >> v1:172.21.15.28:6789/0 conn(0x7fcd0015ba60 legacy=0x7fcd000b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcce40039c0 0x7fcce4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcce4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcce40051d0 0x7fcce400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000cfde0 con 0x7fcce4004810 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400c080 con 0x7fcce40039c0 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400c280 con 0x7fcce40051d0 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcce4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccffe00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcce40039c0 0x7fcce4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd04e00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcce40051d0 0x7fcce400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 -- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcce40051d0 msgr2=0x7fcce400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcce40051d0 0x7fcce400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 -- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcce40039c0 msgr2=0x7fcce4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcce40039c0 0x7fcce4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccff400640 1 -- 172.21.15.28:0/268942047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4003530 con 0x7fcce4004810 2024-03-21T08:44:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fcd04e00640 1 -- 172.21.15.28:0/268942047 reap_dead start 2024-03-21T08:44:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.540+0000 7fccffe00640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcce40039c0 0x7fcce4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fccff400640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcce4004c90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccec01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152140529f8726ac server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0b2070 con 0x7fcce4004810 2024-03-21T08:44:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec0a5660 con 0x7fcce4004810 2024-03-21T08:44:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fccfea00640 1 -- 172.21.15.28:0/268942047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0096b0 con 0x7fcce4004810 2024-03-21T08:44:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 -- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 msgr2=0x7fcce4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcce4004c90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ae0 tx=0x7fccec01ca70 comp rx=0 tx=0).stop 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 -- 172.21.15.28:0/268942047 shutdown_connections 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 --2- 172.21.15.28:0/268942047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcce4004c90 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 -- 172.21.15.28:0/268942047 >> 172.21.15.28:0/268942047 conn(0x7fcd0006fa90 msgr2=0x7fcd0006fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 -- 172.21.15.28:0/268942047 shutdown_connections 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.544+0000 7fcd05800640 1 -- 172.21.15.28:0/268942047 wait complete. 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 Processor -- start 2024-03-21T08:44:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- start start 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcd000af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd001f4e70 0x7fcd000ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd0015d1f0 0x7fcd0015d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015be40 con 0x7fcce4004810 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000d0bc0 con 0x7fcd001f4e70 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015bfc0 con 0x7fcd0015d1f0 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcd000af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcd000af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33300/0 (socket says 172.21.15.28:33300) 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 -- 172.21.15.28:0/2737566787 learned_addr learned my addr 172.21.15.28:0/2737566787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccff400640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd001f4e70 0x7fcd000ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd04e00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd0015d1f0 0x7fcd0015d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 -- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd0015d1f0 msgr2=0x7fcd0015d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fcd0015d1f0 0x7fcd0015d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 -- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd001f4e70 msgr2=0x7fcd000ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fcd001f4e70 0x7fcd000ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000c8bd0 con 0x7fcce4004810 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccffe00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcd000af290 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fccf000e7b0 tx=0x7fccf000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000ee60 con 0x7fcce4004810 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf0018070 con 0x7fcce4004810 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000caf0 con 0x7fcce4004810 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd000c8eb0 con 0x7fcce4004810 2024-03-21T08:44:22.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd000c9470 con 0x7fcce4004810 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.548+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd0004f410 con 0x7fcce4004810 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fccf000cc90 con 0x7fcce4004810 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcce0067b50 0x7fcce006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fccf0086510 con 0x7fcce4004810 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7fcce006dff0 0x7fcce0070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcce0070b00 con 0x7fcce006dff0 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccf00868c0 con 0x7fcce4004810 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccff400640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcce0067b50 0x7fcce006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fcd04e00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7fcce006dff0 0x7fcce0070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fcd04e00640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7fcce006dff0 0x7fcce0070430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccff400640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcce0067b50 0x7fcce006a010 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fccec0af000 tx=0x7fccec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.552+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== osd.3 v2:172.21.15.28:6816/2425819575 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcce0070b00 con 0x7fcce006dff0 2024-03-21T08:44:22.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 --> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcd000b6aa0 con 0x7fcce006dff0 2024-03-21T08:44:22.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fccf7e00640 1 -- 172.21.15.28:0/2737566787 <== osd.3 v2:172.21.15.28:6816/2425819575 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcd000b6aa0 con 0x7fcce006dff0 2024-03-21T08:44:22.567 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:22.568 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:22.568 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7fcce006dff0 msgr2=0x7fcce0070430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:6816/2425819575,v1:172.21.15.28:6817/2425819575] conn(0x7fcce006dff0 0x7fcce0070430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcce0067b50 msgr2=0x7fcce006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fcce0067b50 0x7fcce006a010 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fccec0af000 tx=0x7fccec008040 comp rx=0 tx=0).stop 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 msgr2=0x7fcd000af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 --2- 172.21.15.28:0/2737566787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce4004810 0x7fcd000af290 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fccf000e7b0 tx=0x7fccf000ec70 comp rx=0 tx=0).stop 2024-03-21T08:44:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd04e00640 1 -- 172.21.15.28:0/2737566787 reap_dead start 2024-03-21T08:44:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 shutdown_connections 2024-03-21T08:44:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 >> 172.21.15.28:0/2737566787 conn(0x7fcd0006fa90 msgr2=0x7fcd000b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 shutdown_connections 2024-03-21T08:44:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.568+0000 7fcd05800640 1 -- 172.21.15.28:0/2737566787 wait complete. 2024-03-21T08:44:22.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-21T08:44:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 Processor -- start 2024-03-21T08:44:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 -- start start 2024-03-21T08:44:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ea4072190 0x7f5ea40cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea415bf90 con 0x7f5ea40b4420 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea415b1d0 con 0x7f5ea40b4800 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5eab200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea415b4b0 con 0x7f5ea4072190 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5ea3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ea40b4420 0x7f5ea4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50494/0 (socket says 172.21.15.28:50494) 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5ea9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ea4072190 0x7f5ea40cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.728+0000 7f5ea3e00640 1 -- 172.21.15.28:0/1107179098 learned_addr learned my addr 172.21.15.28:0/1107179098 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea9200640 1 -- 172.21.15.28:0/1107179098 >> v1:172.21.15.151:6789/0 conn(0x7f5ea40b4800 legacy=0x7f5ea40ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea9200640 1 -- 172.21.15.28:0/1107179098 >> v1:172.21.15.28:6789/0 conn(0x7f5ea40b4420 legacy=0x7f5ea4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea9200640 1 -- 172.21.15.28:0/1107179098 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea415b790 con 0x7f5ea4072190 2024-03-21T08:44:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea9200640 1 --2- 172.21.15.28:0/1107179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4072190 0x7f5ea40cd300 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009d90 tx=0x7f5e9c01eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99b9ac527314db4 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea3400640 1 -- 172.21.15.28:0/1107179098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0b4070 con 0x7f5ea4072190 2024-03-21T08:44:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea3400640 1 -- 172.21.15.28:0/1107179098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0a7ac0 con 0x7f5ea4072190 2024-03-21T08:44:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5ea3400640 1 -- 172.21.15.28:0/1107179098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0af440 con 0x7f5ea4072190 2024-03-21T08:44:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5eab200640 1 -- 172.21.15.28:0/1107179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4072190 msgr2=0x7f5ea40cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5eab200640 1 --2- 172.21.15.28:0/1107179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4072190 0x7f5ea40cd300 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009d90 tx=0x7f5e9c01eb50 comp rx=0 tx=0).stop 2024-03-21T08:44:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5eab200640 1 -- 172.21.15.28:0/1107179098 shutdown_connections 2024-03-21T08:44:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5eab200640 1 --2- 172.21.15.28:0/1107179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4072190 0x7f5ea40cd300 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.732+0000 7f5eab200640 1 -- 172.21.15.28:0/1107179098 >> 172.21.15.28:0/1107179098 conn(0x7f5ea40674f0 msgr2=0x7f5ea406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- 172.21.15.28:0/1107179098 shutdown_connections 2024-03-21T08:44:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- 172.21.15.28:0/1107179098 wait complete. 2024-03-21T08:44:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 Processor -- start 2024-03-21T08:44:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- start start 2024-03-21T08:44:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ea4072190 0x7f5ea4144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ea414c840 0x7f5ea414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea406e430 con 0x7f5ea40b4420 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea406e2b0 con 0x7f5ea4072190 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea406e5b0 con 0x7f5ea414c840 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33312/0 (socket says 172.21.15.28:33312) 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 -- 172.21.15.28:0/3255248169 learned_addr learned my addr 172.21.15.28:0/3255248169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ea4072190 0x7f5ea4144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5ea9200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ea414c840 0x7f5ea414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 -- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ea414c840 msgr2=0x7f5ea414e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ea414c840 0x7f5ea414e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 -- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ea4072190 msgr2=0x7f5ea4144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ea4072190 0x7f5ea4144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 -- 172.21.15.28:0/3255248169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea414f460 con 0x7f5ea40b4420 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5e9be00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c3c0 tx=0x7f5e8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c017070 con 0x7f5ea40b4420 2024-03-21T08:44:22.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c0052d0 con 0x7f5ea40b4420 2024-03-21T08:44:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea414f740 con 0x7f5ea40b4420 2024-03-21T08:44:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.736+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c003820 con 0x7f5ea40b4420 2024-03-21T08:44:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea4149320 con 0x7f5ea40b4420 2024-03-21T08:44:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea414cc50 con 0x7f5ea40b4420 2024-03-21T08:44:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5e8c01a070 con 0x7f5ea40b4420 2024-03-21T08:44:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 0x7f5e80069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5e8c0850c0 con 0x7f5ea40b4420 2024-03-21T08:44:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 0x7f5e80069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.740+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 0x7f5e80069fe0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005890 tx=0x7f5e94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.744+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5ea414cc50 con 0x7f5ea40b4420 2024-03-21T08:44:22.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ea4073430 con 0x7f5e80067b20 2024-03-21T08:44:22.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5ea0c00640 1 -- 172.21.15.28:0/3255248169 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ea4073430 con 0x7f5e80067b20 2024-03-21T08:44:22.908 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 msgr2=0x7f5e80069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 0x7f5e80069fe0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005890 tx=0x7f5e94009040 comp rx=0 tx=0).stop 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 msgr2=0x7f5ea414c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c3c0 tx=0x7f5e8c00c880 comp rx=0 tx=0).stop 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 shutdown_connections 2024-03-21T08:44:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5e80067b20 0x7f5e80069fe0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5ea414c840 0x7f5ea414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea40b4420 0x7f5ea414c300 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 --2- 172.21.15.28:0/3255248169 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5ea4072190 0x7f5ea4144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 >> 172.21.15.28:0/3255248169 conn(0x7f5ea40674f0 msgr2=0x7f5ea414df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 shutdown_connections 2024-03-21T08:44:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:22.908+0000 7f5eab200640 1 -- 172.21.15.28:0/3255248169 wait complete. 2024-03-21T08:44:22.929 INFO:tasks.ceph:Scrubbing osd.4 2024-03-21T08:44:22.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.072+0000 7fa2baa00640 1 Processor -- start 2024-03-21T08:44:23.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.072+0000 7fa2baa00640 1 -- start start 2024-03-21T08:44:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2baa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 0x7fa2b40d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2b40cf9b0 0x7fa2b40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2baa00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40d0040 con 0x7fa2b415ba60 2024-03-21T08:44:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2baa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40cfe00 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2baa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40d0320 con 0x7fa2b40b0f60 2024-03-21T08:44:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2b40cf9b0 0x7fa2b40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2b40cf9b0 0x7fa2b40ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33316/0 (socket says 172.21.15.28:33316) 2024-03-21T08:44:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 -- 172.21.15.28:0/3685321211 learned_addr learned my addr 172.21.15.28:0/3685321211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 0x7fa2b40d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 -- 172.21.15.28:0/3685321211 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 msgr2=0x7fa2b40d10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 0x7fa2b40d10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 -- 172.21.15.28:0/3685321211 >> v1:172.21.15.151:6789/0 conn(0x7fa2b415ba60 legacy=0x7fa2b40b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2ba000640 1 -- 172.21.15.28:0/3685321211 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40d0600 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.076+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 0x7fa2b40d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2ba000640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40cf9b0 0x7fa2b40ce820 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009e50 tx=0x7fa2ac01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75607c81cd9c4439 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2b3e00640 1 -- 172.21.15.28:0/3685321211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac0b4070 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2b3e00640 1 -- 172.21.15.28:0/3685321211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac0af400 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2b3e00640 1 -- 172.21.15.28:0/3685321211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac0af7d0 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- 172.21.15.28:0/3685321211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40cf9b0 msgr2=0x7fa2b40ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40cf9b0 0x7fa2b40ce820 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009e50 tx=0x7fa2ac01f210 comp rx=0 tx=0).stop 2024-03-21T08:44:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- 172.21.15.28:0/3685321211 shutdown_connections 2024-03-21T08:44:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40cf9b0 0x7fa2b40ce820 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- 172.21.15.28:0/3685321211 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40b0f60 0x7fa2b40d10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- 172.21.15.28:0/3685321211 >> 172.21.15.28:0/3685321211 conn(0x7fa2b406fa90 msgr2=0x7fa2b406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- 172.21.15.28:0/3685321211 shutdown_connections 2024-03-21T08:44:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- 172.21.15.28:0/3685321211 wait complete. 2024-03-21T08:44:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 Processor -- start 2024-03-21T08:44:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- start start 2024-03-21T08:44:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 0x7fa2b40cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40cf9b0 0x7fa2b4159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2b415ba60 0x7fa2b40c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b415be40 con 0x7fa2b40b0f60 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40d0bc0 con 0x7fa2b415ba60 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.080+0000 7fa2baa00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b415bfc0 con 0x7fa2b40cf9b0 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 0x7fa2b40cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 0x7fa2b40cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33318/0 (socket says 172.21.15.28:33318) 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 -- 172.21.15.28:0/1590443090 learned_addr learned my addr 172.21.15.28:0/1590443090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2ba000640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2b415ba60 0x7fa2b40c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40cf9b0 0x7fa2b4159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 -- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40cf9b0 msgr2=0x7fa2b4159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40cf9b0 0x7fa2b4159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 -- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2b415ba60 msgr2=0x7fa2b40c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa2b415ba60 0x7fa2b40c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40c98c0 con 0x7fa2b40b0f60 2024-03-21T08:44:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa2b40cf9b0 0x7fa2b4159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b9600640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 0x7fa2b40cb640 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c3a0 tx=0x7fa2a800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80052d0 con 0x7fa2b40b0f60 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8005470 con 0x7fa2b40b0f60 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8007260 con 0x7fa2b40b0f60 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b40c9b70 con 0x7fa2b40b0f60 2024-03-21T08:44:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b406c3b0 con 0x7fa2b40b0f60 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2b404f410 con 0x7fa2b40b0f60 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa2a8007400 con 0x7fa2b40b0f60 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.084+0000 7fa2b1600640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa294067b20 0x7fa294069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa2a8085d00 con 0x7fa2b40b0f60 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b1600640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7fa29406dfc0 0x7fa294070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2a8009e50 con 0x7fa29406dfc0 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2a80860b0 con 0x7fa2b40b0f60 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa294067b20 0x7fa294069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2ba000640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7fa29406dfc0 0x7fa294070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b8c00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa294067b20 0x7fa294069fe0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40cee60 tx=0x7fa2a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2ba000640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7fa29406dfc0 0x7fa294070400 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.088+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== osd.4 v2:172.21.15.151:6809/3622853820 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa2a8009e50 con 0x7fa29406dfc0 2024-03-21T08:44:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.100+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 --> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa2b40d1330 con 0x7fa29406dfc0 2024-03-21T08:44:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2b1600640 1 -- 172.21.15.28:0/1590443090 <== osd.4 v2:172.21.15.151:6809/3622853820 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa2b40d1330 con 0x7fa29406dfc0 2024-03-21T08:44:23.103 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:23.103 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:23.103 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7fa29406dfc0 msgr2=0x7fa294070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6809/3622853820,v1:172.21.15.151:6811/3622853820] conn(0x7fa29406dfc0 0x7fa294070400 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa294067b20 msgr2=0x7fa294069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa294067b20 0x7fa294069fe0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40cee60 tx=0x7fa2a0008040 comp rx=0 tx=0).stop 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 msgr2=0x7fa2b40cb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 --2- 172.21.15.28:0/1590443090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2b40b0f60 0x7fa2b40cb640 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c3a0 tx=0x7fa2a800c860 comp rx=0 tx=0).stop 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2ba000640 1 -- 172.21.15.28:0/1590443090 reap_dead start 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 shutdown_connections 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 >> 172.21.15.28:0/1590443090 conn(0x7fa2b406fa90 msgr2=0x7fa2b40b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 shutdown_connections 2024-03-21T08:44:23.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.104+0000 7fa2baa00640 1 -- 172.21.15.28:0/1590443090 wait complete. 2024-03-21T08:44:23.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-21T08:44:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 Processor -- start 2024-03-21T08:44:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 -- start start 2024-03-21T08:44:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 0x7feef406cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7feef40d0fd0 0x7feef40ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef40b48a0 con 0x7feef406d1d0 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef406d630 con 0x7feef40d0fd0 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef9c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40cc0a0 con 0x7feef415df10 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feeea000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feef406d1d0 0x7feef40d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50510/0 (socket says 172.21.15.28:50510) 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feeea000640 1 -- 172.21.15.28:0/3556517477 learned_addr learned my addr 172.21.15.28:0/3556517477 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2a00640 1 --2- 172.21.15.28:0/3556517477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 0x7feef406cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef3400640 1 --2- 172.21.15.28:0/3556517477 >> v2:172.21.15.151:3300/0 conn(0x7feef40d0fd0 0x7feef40ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2000640 1 -- 172.21.15.28:0/3556517477 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406283647 0 0) 0x7feedc001120 con 0x7feef406d1d0 2024-03-21T08:44:23.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2000640 1 -- 172.21.15.28:0/3556517477 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed8003530 con 0x7feef406d1d0 2024-03-21T08:44:23.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2a00640 1 -- 172.21.15.28:0/3556517477 >> v2:172.21.15.151:3300/0 conn(0x7feef40d0fd0 msgr2=0x7feef40ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:23.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2a00640 1 --2- 172.21.15.28:0/3556517477 >> v2:172.21.15.151:3300/0 conn(0x7feef40d0fd0 0x7feef40ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2a00640 1 -- 172.21.15.28:0/3556517477 >> v1:172.21.15.28:6789/0 conn(0x7feef406d1d0 legacy=0x7feef40d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.264+0000 7feef2a00640 1 -- 172.21.15.28:0/3556517477 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40d1620 con 0x7feef415df10 2024-03-21T08:44:23.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef2a00640 1 --2- 172.21.15.28:0/3556517477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 0x7feef406cc90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feee40097c0 tx=0x7feee401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4670a47c80bc9057 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef2000640 1 -- 172.21.15.28:0/3556517477 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee40b0070 con 0x7feef415df10 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef2000640 1 -- 172.21.15.28:0/3556517477 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee40ab600 con 0x7feef415df10 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef2000640 1 -- 172.21.15.28:0/3556517477 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee40ab9d0 con 0x7feef415df10 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 -- 172.21.15.28:0/3556517477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 msgr2=0x7feef406cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 --2- 172.21.15.28:0/3556517477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 0x7feef406cc90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feee40097c0 tx=0x7feee401c410 comp rx=0 tx=0).stop 2024-03-21T08:44:23.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 -- 172.21.15.28:0/3556517477 shutdown_connections 2024-03-21T08:44:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 --2- 172.21.15.28:0/3556517477 >> v2:172.21.15.151:3300/0 conn(0x7feef40d0fd0 0x7feef40ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 --2- 172.21.15.28:0/3556517477 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef415df10 0x7feef406cc90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 -- 172.21.15.28:0/3556517477 >> 172.21.15.28:0/3556517477 conn(0x7feef40b3540 msgr2=0x7feef40b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 -- 172.21.15.28:0/3556517477 shutdown_connections 2024-03-21T08:44:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.268+0000 7feef9c00640 1 -- 172.21.15.28:0/3556517477 wait complete. 2024-03-21T08:44:23.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 Processor -- start 2024-03-21T08:44:23.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- start start 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef40d0fd0 0x7feef414f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef415df10 0x7feef415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef40cdeb0 con 0x7feef406d1d0 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef40cdd30 con 0x7feef415df10 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40cc940 con 0x7feef40d0fd0 2024-03-21T08:44:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33326/0 (socket says 172.21.15.28:33326) 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 -- 172.21.15.28:0/3842262993 learned_addr learned my addr 172.21.15.28:0/3842262993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feeea000640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef40d0fd0 0x7feef414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef3400640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef415df10 0x7feef415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 -- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef40d0fd0 msgr2=0x7feef414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef40d0fd0 0x7feef414f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 -- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef415df10 msgr2=0x7feef415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef415df10 0x7feef415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 -- 172.21.15.28:0/3842262993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4151320 con 0x7feef406d1d0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef2a00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7feee40ae040 tx=0x7feee401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee40b0050 con 0x7feef406d1d0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee40b1040 con 0x7feef406d1d0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4006e20 con 0x7feef406d1d0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef414fd80 con 0x7feef406d1d0 2024-03-21T08:44:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.272+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef4158840 con 0x7feef406d1d0 2024-03-21T08:44:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef404f4f0 con 0x7feef406d1d0 2024-03-21T08:44:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7feee40be030 con 0x7feef406d1d0 2024-03-21T08:44:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feeeb400640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 0x7feed0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7feee411fa80 con 0x7feef406d1d0 2024-03-21T08:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feeea000640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 0x7feed0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.276+0000 7feeea000640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 0x7feed0069fe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7feef41548e0 tx=0x7feedc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.280+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feee4121e20 con 0x7feef406d1d0 2024-03-21T08:44:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.444+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7feef406fef0 con 0x7feed0067b20 2024-03-21T08:44:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.444+0000 7feeeb400640 1 -- 172.21.15.28:0/3842262993 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feef406fef0 con 0x7feed0067b20 2024-03-21T08:44:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-03-21T08:44:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 msgr2=0x7feed0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 0x7feed0069fe0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7feef41548e0 tx=0x7feedc009040 comp rx=0 tx=0).stop 2024-03-21T08:44:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 msgr2=0x7feef4157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7feee40ae040 tx=0x7feee401ac50 comp rx=0 tx=0).stop 2024-03-21T08:44:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 shutdown_connections 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7feef415df10 0x7feef415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7feed0067b20 0x7feed0069fe0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7feef40d0fd0 0x7feef414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 --2- 172.21.15.28:0/3842262993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef406d1d0 0x7feef4157ed0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 >> 172.21.15.28:0/3842262993 conn(0x7feef40b3540 msgr2=0x7feef415a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 shutdown_connections 2024-03-21T08:44:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.448+0000 7feef9c00640 1 -- 172.21.15.28:0/3842262993 wait complete. 2024-03-21T08:44:23.467 INFO:tasks.ceph:Scrubbing osd.5 2024-03-21T08:44:23.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 Processor -- start 2024-03-21T08:44:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 -- start start 2024-03-21T08:44:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02d40cd2b0 0x7f02d40b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d415bfd0 con 0x7f02d4071f60 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d40cd8d0 con 0x7f02d40cd2b0 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02da600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d415b230 con 0x7f02d40b43e0 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f02d40cd2b0 0x7f02d40b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:56198/0 (socket says 172.21.15.28:56198) 2024-03-21T08:44:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 -- 172.21.15.28:0/2720787692 learned_addr learned my addr 172.21.15.28:0/2720787692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02cbe00640 1 --1- 172.21.15.28:0/2720787692 >> v1:172.21.15.151:6789/0 conn(0x7f02d4071f60 0x7f02d40ccba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42046/0 (socket says 172.21.15.28:42046) 2024-03-21T08:44:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 -- 172.21.15.28:0/2720787692 >> v2:172.21.15.151:3300/0 conn(0x7f02d40cd2b0 msgr2=0x7f02d40b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 --2- 172.21.15.28:0/2720787692 >> v2:172.21.15.151:3300/0 conn(0x7f02d40cd2b0 0x7f02d40b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 -- 172.21.15.28:0/2720787692 >> v1:172.21.15.151:6789/0 conn(0x7f02d4071f60 legacy=0x7f02d40ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.612+0000 7f02d3400640 1 -- 172.21.15.28:0/2720787692 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d415b4f0 con 0x7f02d40b43e0 2024-03-21T08:44:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02d3400640 1 --2- 172.21.15.28:0/2720787692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009e50 tx=0x7f02c401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=118fcc7fa936af05 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02d2a00640 1 -- 172.21.15.28:0/2720787692 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40a5700 con 0x7f02d40b43e0 2024-03-21T08:44:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02d2a00640 1 -- 172.21.15.28:0/2720787692 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c40a58a0 con 0x7f02d40b43e0 2024-03-21T08:44:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02d2a00640 1 -- 172.21.15.28:0/2720787692 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40a5c70 con 0x7f02d40b43e0 2024-03-21T08:44:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 -- 172.21.15.28:0/2720787692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 msgr2=0x7f02d4071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 --2- 172.21.15.28:0/2720787692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009e50 tx=0x7f02c401ca30 comp rx=0 tx=0).stop 2024-03-21T08:44:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 -- 172.21.15.28:0/2720787692 shutdown_connections 2024-03-21T08:44:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 --2- 172.21.15.28:0/2720787692 >> v2:172.21.15.151:3300/0 conn(0x7f02d40cd2b0 0x7f02d40b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 --2- 172.21.15.28:0/2720787692 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d4071a20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 -- 172.21.15.28:0/2720787692 >> 172.21.15.28:0/2720787692 conn(0x7f02d40674f0 msgr2=0x7f02d406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 -- 172.21.15.28:0/2720787692 shutdown_connections 2024-03-21T08:44:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.616+0000 7f02da600640 1 -- 172.21.15.28:0/2720787692 wait complete. 2024-03-21T08:44:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 Processor -- start 2024-03-21T08:44:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 -- start start 2024-03-21T08:44:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 0x7f02d41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d41530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d40cd2b0 0x7f02d41535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4072360 con 0x7f02d40cd2b0 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b49d0 con 0x7f02d4071f60 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02da600640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b37b0 con 0x7f02d40b43e0 2024-03-21T08:44:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 0x7f02d41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d40cd2b0 0x7f02d41535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 0x7f02d41650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:54616/0 (socket says 172.21.15.28:54616) 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 -- 172.21.15.28:0/3991452910 learned_addr learned my addr 172.21.15.28:0/3991452910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02cbe00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d41530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 -- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 msgr2=0x7f02d41530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f02d40b43e0 0x7f02d41530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 -- 172.21.15.28:0/3991452910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d40cd2b0 msgr2=0x7f02d41535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d40cd2b0 0x7f02d41535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4153d00 con 0x7f02d4071f60 2024-03-21T08:44:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3e00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d40cd2b0 0x7f02d41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02d3400640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 0x7f02d41650d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02c401a660 tx=0x7f02c40197d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.620+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40b8670 con 0x7f02d4071f60 2024-03-21T08:44:23.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c40be070 con 0x7f02d4071f60 2024-03-21T08:44:23.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d4153fe0 con 0x7f02d4071f60 2024-03-21T08:44:23.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40b8a70 con 0x7f02d4071f60 2024-03-21T08:44:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41545a0 con 0x7f02d4071f60 2024-03-21T08:44:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02d4068110 con 0x7f02d4071f60 2024-03-21T08:44:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f02c40b8c10 con 0x7f02d4071f60 2024-03-21T08:44:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f02b0067b20 0x7f02b0069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f02c4121740 con 0x7f02d4071f60 2024-03-21T08:44:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7f02b006dfc0 0x7f02b0070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02c40b7e50 con 0x7f02b006dfc0 2024-03-21T08:44:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== mon.1 v2:172.21.15.151:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c4121bd0 con 0x7f02d4071f60 2024-03-21T08:44:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02cbe00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f02b0067b20 0x7f02b0069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.624+0000 7f02d3e00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7f02b006dfc0 0x7f02b0070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.628+0000 7f02cbe00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f02b0067b20 0x7f02b0069fe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f02d4130be0 tx=0x7f02bc007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.628+0000 7f02d3e00640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7f02b006dfc0 0x7f02b0070400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.628+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== osd.5 v2:172.21.15.151:6800/2309128518 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02c40b7e50 con 0x7f02b006dfc0 2024-03-21T08:44:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 --> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f02d41476d0 con 0x7f02b006dfc0 2024-03-21T08:44:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02cb400640 1 -- 172.21.15.28:0/3991452910 <== osd.5 v2:172.21.15.151:6800/2309128518 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f02d41476d0 con 0x7f02b006dfc0 2024-03-21T08:44:23.642 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:23.642 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:23.642 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7f02b006dfc0 msgr2=0x7f02b0070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6800/2309128518,v1:172.21.15.151:6801/2309128518] conn(0x7f02b006dfc0 0x7f02b0070400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f02b0067b20 msgr2=0x7f02b0069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f02b0067b20 0x7f02b0069fe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f02d4130be0 tx=0x7f02bc007e90 comp rx=0 tx=0).stop 2024-03-21T08:44:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 msgr2=0x7f02d41650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 --2- 172.21.15.28:0/3991452910 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f02d4071f60 0x7f02d41650d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02c401a660 tx=0x7f02c40197d0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02d3e00640 1 -- 172.21.15.28:0/3991452910 reap_dead start 2024-03-21T08:44:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 shutdown_connections 2024-03-21T08:44:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 >> 172.21.15.28:0/3991452910 conn(0x7f02d40674f0 msgr2=0x7f02d4151100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 shutdown_connections 2024-03-21T08:44:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.640+0000 7f02da600640 1 -- 172.21.15.28:0/3991452910 wait complete. 2024-03-21T08:44:23.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-21T08:44:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 Processor -- start 2024-03-21T08:44:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 -- start start 2024-03-21T08:44:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792015e020 0x7f79200cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f792006b880 0x7f79200ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79200d0760 con 0x7f79200ccb70 2024-03-21T08:44:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200ce7b0 con 0x7f792006b880 2024-03-21T08:44:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f7925c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79200cfc90 con 0x7f792015e020 2024-03-21T08:44:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791ea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792015e020 0x7f79200cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f792006b880 0x7f79200ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791e000640 1 --1- >> v1:172.21.15.151:6789/0 conn(0x7f79200ccb70 0x7f792006b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42056/0 (socket says 172.21.15.28:42056) 2024-03-21T08:44:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f792006b880 0x7f79200ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:54626/0 (socket says 172.21.15.28:54626) 2024-03-21T08:44:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791e000640 1 -- 172.21.15.28:0/1605326049 learned_addr learned my addr 172.21.15.28:0/1605326049 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 -- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792015e020 msgr2=0x7f79200cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 --2- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792015e020 0x7f79200cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 -- 172.21.15.28:0/1605326049 >> v1:172.21.15.151:6789/0 conn(0x7f79200ccb70 legacy=0x7f792006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791f400640 1 -- 172.21.15.28:0/1605326049 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200cfec0 con 0x7f792006b880 2024-03-21T08:44:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.804+0000 7f791d600640 1 -- 172.21.15.28:0/1605326049 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263163749 0 0) 0x7f79200d0760 con 0x7f79200ccb70 2024-03-21T08:44:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f791f400640 1 --2- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792006b880 0x7f79200ccfd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7918009d90 tx=0x7f791801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc06a529e0739de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f791d600640 1 -- 172.21.15.28:0/1605326049 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180b2070 con 0x7f792006b880 2024-03-21T08:44:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f791d600640 1 -- 172.21.15.28:0/1605326049 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79180a5920 con 0x7f792006b880 2024-03-21T08:44:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f791d600640 1 -- 172.21.15.28:0/1605326049 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180a5cf0 con 0x7f792006b880 2024-03-21T08:44:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 -- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792006b880 msgr2=0x7f79200ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 --2- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792006b880 0x7f79200ccfd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7918009d90 tx=0x7f791801c970 comp rx=0 tx=0).stop 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 -- 172.21.15.28:0/1605326049 shutdown_connections 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 --2- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792006b880 0x7f79200ccfd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 --2- 172.21.15.28:0/1605326049 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792015e020 0x7f79200cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 -- 172.21.15.28:0/1605326049 >> 172.21.15.28:0/1605326049 conn(0x7f7920099a30 msgr2=0x7f7920099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 -- 172.21.15.28:0/1605326049 shutdown_connections 2024-03-21T08:44:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.808+0000 7f7925c00640 1 -- 172.21.15.28:0/1605326049 wait complete. 2024-03-21T08:44:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 Processor -- start 2024-03-21T08:44:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- start start 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792006b880 0x7f792006f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792015e020 0x7f7920153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792015f410 con 0x7f79200ccb70 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79200cb590 con 0x7f792015e020 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792015f590 con 0x7f792006b880 2024-03-21T08:44:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33342/0 (socket says 172.21.15.28:33342) 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 -- 172.21.15.28:0/3660811275 learned_addr learned my addr 172.21.15.28:0/3660811275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791f400640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792015e020 0x7f7920153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791ea00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792006b880 0x7f792006f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 -- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792006b880 msgr2=0x7f792006f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792006b880 0x7f792006f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 -- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792015e020 msgr2=0x7f7920153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792015e020 0x7f7920153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 -- 172.21.15.28:0/3660811275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79201589a0 con 0x7f79200ccb70 2024-03-21T08:44:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f791e000640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f790c00c520 tx=0x7f790c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c018070 con 0x7f79200ccb70 2024-03-21T08:44:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c0074e0 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7920157320 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790c0049f0 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79201578e0 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.812+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792009eb60 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.816+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f790c01b070 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.816+0000 7f7916a00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 0x7f7900069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.816+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f790c085200 con 0x7f79200ccb70 2024-03-21T08:44:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.816+0000 7f791ea00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 0x7f7900069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.816+0000 7f791ea00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 0x7f7900069fe0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7920070bf0 tx=0x7f7910009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.820+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f79201589a0 con 0x7f79200ccb70 2024-03-21T08:44:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f792004f4f0 con 0x7f7900067b20 2024-03-21T08:44:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7916a00640 1 -- 172.21.15.28:0/3660811275 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f792004f4f0 con 0x7f7900067b20 2024-03-21T08:44:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-03-21T08:44:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 msgr2=0x7f7900069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 0x7f7900069fe0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7920070bf0 tx=0x7f7910009190 comp rx=0 tx=0).stop 2024-03-21T08:44:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 msgr2=0x7f7920152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f790c00c520 tx=0x7f790c00c9e0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 shutdown_connections 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f792015e020 0x7f7920153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79200ccb70 0x7f7920152be0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f7900067b20 0x7f7900069fe0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 --2- 172.21.15.28:0/3660811275 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f792006b880 0x7f792006f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 >> 172.21.15.28:0/3660811275 conn(0x7f7920099a30 msgr2=0x7f792015b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 shutdown_connections 2024-03-21T08:44:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:23.984+0000 7f7925c00640 1 -- 172.21.15.28:0/3660811275 wait complete. 2024-03-21T08:44:24.005 INFO:tasks.ceph:Scrubbing osd.6 2024-03-21T08:44:24.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:24.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.148+0000 7f5ce2e00640 1 Processor -- start 2024-03-21T08:44:24.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.148+0000 7f5ce2e00640 1 -- start start 2024-03-21T08:44:24.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce2e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cdc072130 0x7f5cdc0ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce2e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5cdc0cd0e0 0x7f5cdc0b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc15bfd0 con 0x7f5cdc0b43e0 2024-03-21T08:44:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce2e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc0cd8d0 con 0x7f5cdc072130 2024-03-21T08:44:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce2e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc15b230 con 0x7f5cdc0cd0e0 2024-03-21T08:44:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cdbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5cdc0b43e0 0x7f5cdc071a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50520/0 (socket says 172.21.15.28:50520) 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cdbe00640 1 -- 172.21.15.28:0/1849483775 learned_addr learned my addr 172.21.15.28:0/1849483775 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 --2- 172.21.15.28:0/1849483775 >> v2:172.21.15.28:3300/0 conn(0x7f5cdc072130 0x7f5cdc0ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5ce0e00640 1 --2- 172.21.15.28:0/1849483775 >> v2:172.21.15.151:3300/0 conn(0x7f5cdc0cd0e0 0x7f5cdc0b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cdb400640 1 -- 172.21.15.28:0/1849483775 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423479884 0 0) 0x7f5cdc15bfd0 con 0x7f5cdc0b43e0 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cdb400640 1 -- 172.21.15.28:0/1849483775 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc0003530 con 0x7f5cdc0b43e0 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 -- 172.21.15.28:0/1849483775 >> v2:172.21.15.151:3300/0 conn(0x7f5cdc0cd0e0 msgr2=0x7f5cdc0b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 --2- 172.21.15.28:0/1849483775 >> v2:172.21.15.151:3300/0 conn(0x7f5cdc0cd0e0 0x7f5cdc0b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 -- 172.21.15.28:0/1849483775 >> v1:172.21.15.28:6789/0 conn(0x7f5cdc0b43e0 legacy=0x7f5cdc071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 -- 172.21.15.28:0/1849483775 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc15b4f0 con 0x7f5cdc072130 2024-03-21T08:44:24.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.152+0000 7f5cd3400640 1 --2- 172.21.15.28:0/1849483775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc0ccba0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06f2a0 tx=0x7f5cc401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2acc1674d03a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdb400640 1 -- 172.21.15.28:0/1849483775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40b0070 con 0x7f5cdc072130 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdb400640 1 -- 172.21.15.28:0/1849483775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc40ab600 con 0x7f5cdc072130 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdb400640 1 -- 172.21.15.28:0/1849483775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40ab9d0 con 0x7f5cdc072130 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- 172.21.15.28:0/1849483775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 msgr2=0x7f5cdc0ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/1849483775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc0ccba0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06f2a0 tx=0x7f5cc401c480 comp rx=0 tx=0).stop 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- 172.21.15.28:0/1849483775 shutdown_connections 2024-03-21T08:44:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/1849483775 >> v2:172.21.15.151:3300/0 conn(0x7f5cdc0cd0e0 0x7f5cdc0b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/1849483775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc0ccba0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- 172.21.15.28:0/1849483775 >> 172.21.15.28:0/1849483775 conn(0x7f5cdc0674f0 msgr2=0x7f5cdc06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- 172.21.15.28:0/1849483775 shutdown_connections 2024-03-21T08:44:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- 172.21.15.28:0/1849483775 wait complete. 2024-03-21T08:44:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 Processor -- start 2024-03-21T08:44:24.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- start start 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc15d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cdc0b43e0 0x7f5cdc152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cdc0cd0e0 0x7f5cdc14c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0b4940 con 0x7f5cdc072130 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0b47c0 con 0x7f5cdc0cd0e0 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce2e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc0b37b0 con 0x7f5cdc0b43e0 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc15d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33366/0 (socket says 172.21.15.28:33366) 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdbe00640 1 -- 172.21.15.28:0/841175357 learned_addr learned my addr 172.21.15.28:0/841175357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cd3400640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cdc0b43e0 0x7f5cdc152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5ce0e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cdc0cd0e0 0x7f5cdc14c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.156+0000 7f5cdbe00640 1 -- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cdc0b43e0 msgr2=0x7f5cdc152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cdbe00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cdc0b43e0 0x7f5cdc152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cdbe00640 1 -- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cdc0cd0e0 msgr2=0x7f5cdc14c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cdbe00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cdc0cd0e0 0x7f5cdc14c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cdbe00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc14b930 con 0x7f5cdc072130 2024-03-21T08:44:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5ce0e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cdc0cd0e0 0x7f5cdc14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cdbe00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc15d250 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c480 tx=0x7f5ccc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc017070 con 0x7f5cdc072130 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc005cf0 con 0x7f5cdc072130 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc007290 con 0x7f5cdc072130 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc14bc10 con 0x7f5cdc072130 2024-03-21T08:44:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc14c1d0 con 0x7f5cdc072130 2024-03-21T08:44:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5cdc068110 con 0x7f5cdc072130 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5ccc01a070 con 0x7f5cdc072130 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b20 0x7f5cb8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5ccc0851d0 con 0x7f5cdc072130 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5cb806dfc0 0x7f5cb8070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ccc016ec0 con 0x7f5cb806dfc0 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ccc08c050 con 0x7f5cdc072130 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5cd3400640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b20 0x7f5cb8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.160+0000 7f5ce0e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5cb806dfc0 0x7f5cb8070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.164+0000 7f5cd3400640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b20 0x7f5cb8069fe0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40ad000 tx=0x7f5cc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.164+0000 7f5ce0e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5cb806dfc0 0x7f5cb8070400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.164+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== osd.6 v2:172.21.15.151:6808/3306491257 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ccc016ec0 con 0x7f5cb806dfc0 2024-03-21T08:44:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.176+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 --> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5cdc06ec40 con 0x7f5cb806dfc0 2024-03-21T08:44:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.176+0000 7f5cd8c00640 1 -- 172.21.15.28:0/841175357 <== osd.6 v2:172.21.15.151:6808/3306491257 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5cdc06ec40 con 0x7f5cb806dfc0 2024-03-21T08:44:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:24.179 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5cb806dfc0 msgr2=0x7f5cb8070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6808/3306491257,v1:172.21.15.151:6810/3306491257] conn(0x7f5cb806dfc0 0x7f5cb8070400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b20 msgr2=0x7f5cb8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b20 0x7f5cb8069fe0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40ad000 tx=0x7f5cc4008040 comp rx=0 tx=0).stop 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 msgr2=0x7f5cdc15d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 --2- 172.21.15.28:0/841175357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cdc072130 0x7f5cdc15d250 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c480 tx=0x7f5ccc00c940 comp rx=0 tx=0).stop 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce0e00640 1 -- 172.21.15.28:0/841175357 reap_dead start 2024-03-21T08:44:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 shutdown_connections 2024-03-21T08:44:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 >> 172.21.15.28:0/841175357 conn(0x7f5cdc0674f0 msgr2=0x7f5cdc073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 shutdown_connections 2024-03-21T08:44:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.180+0000 7f5ce2e00640 1 -- 172.21.15.28:0/841175357 wait complete. 2024-03-21T08:44:24.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-21T08:44:24.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 Processor -- start 2024-03-21T08:44:24.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 -- start start 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 0x7f04a80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04a80cc170 0x7f04a8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a815be30 con 0x7f04a80ce320 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a80ccea0 con 0x7f04a80cc170 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a80cc8e0 con 0x7f04a815ba50 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f049e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04a80cc170 0x7f04a8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f04a80ce320 0x7f04a80b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50522/0 (socket says 172.21.15.28:50522) 2024-03-21T08:44:24.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a7400640 1 -- 172.21.15.28:0/11791168 learned_addr learned my addr 172.21.15.28:0/11791168 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:24.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a6a00640 1 --2- 172.21.15.28:0/11791168 >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 0x7f04a80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f049e000640 1 -- 172.21.15.28:0/11791168 >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 msgr2=0x7f04a80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f049e000640 1 --2- 172.21.15.28:0/11791168 >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 0x7f04a80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f049e000640 1 -- 172.21.15.28:0/11791168 >> v1:172.21.15.28:6789/0 conn(0x7f04a80ce320 legacy=0x7f04a80b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f049e000640 1 -- 172.21.15.28:0/11791168 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a80ccbc0 con 0x7f04a80cc170 2024-03-21T08:44:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a6000640 1 -- 172.21.15.28:0/11791168 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020687133 0 0) 0x7f04a815be30 con 0x7f04a80ce320 2024-03-21T08:44:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.340+0000 7f04a6a00640 1 --2- 172.21.15.28:0/11791168 >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 0x7f04a80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:24.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f049e000640 1 --2- 172.21.15.28:0/11791168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80cc170 0x7f04a8071bb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f04a80cf100 tx=0x7f049001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf4eb16531b369b server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a6000640 1 -- 172.21.15.28:0/11791168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900ab650 con 0x7f04a80cc170 2024-03-21T08:44:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a6000640 1 -- 172.21.15.28:0/11791168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04900b0070 con 0x7f04a80cc170 2024-03-21T08:44:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a6000640 1 -- 172.21.15.28:0/11791168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900aba10 con 0x7f04a80cc170 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 -- 172.21.15.28:0/11791168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80cc170 msgr2=0x7f04a8071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 --2- 172.21.15.28:0/11791168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80cc170 0x7f04a8071bb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f04a80cf100 tx=0x7f049001c450 comp rx=0 tx=0).stop 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 -- 172.21.15.28:0/11791168 shutdown_connections 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 --2- 172.21.15.28:0/11791168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80cc170 0x7f04a8071bb0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 --2- 172.21.15.28:0/11791168 >> v2:172.21.15.151:3300/0 conn(0x7f04a815ba50 0x7f04a80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 -- 172.21.15.28:0/11791168 >> 172.21.15.28:0/11791168 conn(0x7f04a806fa90 msgr2=0x7f04a806fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 -- 172.21.15.28:0/11791168 shutdown_connections 2024-03-21T08:44:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.344+0000 7f04a7e00640 1 -- 172.21.15.28:0/11791168 wait complete. 2024-03-21T08:44:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 Processor -- start 2024-03-21T08:44:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- start start 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 0x7f04a8165f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 0x7f04a80c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80720f0 con 0x7f04a80ce320 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80cc5e0 con 0x7f04a815ba50 2024-03-21T08:44:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80ceb40 con 0x7f04a80cc170 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33386/0 (socket says 172.21.15.28:33386) 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 -- 172.21.15.28:0/3162870992 learned_addr learned my addr 172.21.15.28:0/3162870992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7400640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 0x7f04a80c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a6a00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 0x7f04a8165f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 -- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 msgr2=0x7f04a8165f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 0x7f04a8165f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 -- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 msgr2=0x7f04a80c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 0x7f04a80c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 -- 172.21.15.28:0/3162870992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a81641b0 con 0x7f04a80ce320 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a6a00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 0x7f04a8165f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7400640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 0x7f04a80c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049e000640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f04900ad040 tx=0x7f049001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900bf610 con 0x7f04a80ce320 2024-03-21T08:44:24.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04900bf7b0 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a80c6460 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900bfba0 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.348+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a80c6a20 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a804f410 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f04900bd020 con 0x7f04a80ce320 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f049f400640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 0x7f0484069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0490120790 con 0x7f04a80ce320 2024-03-21T08:44:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f04a6a00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 0x7f0484069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.352+0000 7f04a6a00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 0x7f0484069fe0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f049800e550 tx=0x7f0498007ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.356+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0490016070 con 0x7f04a80ce320 2024-03-21T08:44:24.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.520+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f04a80c6c60 con 0x7f0484067b20 2024-03-21T08:44:24.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.520+0000 7f049f400640 1 -- 172.21.15.28:0/3162870992 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f049800c5c0 con 0x7f0484067b20 2024-03-21T08:44:24.522 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 msgr2=0x7f0484069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 0x7f0484069fe0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f049800e550 tx=0x7f0498007ab0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 msgr2=0x7f04a8163a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f04900ad040 tx=0x7f049001ac90 comp rx=0 tx=0).stop 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 shutdown_connections 2024-03-21T08:44:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f04a815ba50 0x7f04a80c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a80ce320 0x7f04a8163a60 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f0484067b20 0x7f0484069fe0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 --2- 172.21.15.28:0/3162870992 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f04a80cc170 0x7f04a8165f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 >> 172.21.15.28:0/3162870992 conn(0x7f04a806fa90 msgr2=0x7f04a80b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 shutdown_connections 2024-03-21T08:44:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.524+0000 7f04a7e00640 1 -- 172.21.15.28:0/3162870992 wait complete. 2024-03-21T08:44:24.544 INFO:tasks.ceph:Scrubbing osd.7 2024-03-21T08:44:24.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-21T08:44:24.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.688+0000 7fa668e00640 1 Processor -- start 2024-03-21T08:44:24.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.688+0000 7fa668e00640 1 -- start start 2024-03-21T08:44:24.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa668e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa6640b43e0 0x7fa664071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa668e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa668e00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66415bfd0 con 0x7fa6640cd0e0 2024-03-21T08:44:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa668e00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6640cd8d0 con 0x7fa6640b43e0 2024-03-21T08:44:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa668e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66415b230 con 0x7fa664071f60 2024-03-21T08:44:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa6640b43e0 0x7fa664071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa65be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7fa6640b43e0 0x7fa664071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3300/0 says I am v2:172.21.15.28:54698/0 (socket says 172.21.15.28:54698) 2024-03-21T08:44:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa65be00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.151:3301/0 says I am v2:172.21.15.28:56268/0 (socket says 172.21.15.28:56268) 2024-03-21T08:44:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 -- 172.21.15.28:0/978827420 learned_addr learned my addr 172.21.15.28:0/978827420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa662800640 1 --1- 172.21.15.28:0/978827420 >> v1:172.21.15.151:6789/0 conn(0x7fa6640cd0e0 0x7fa66415e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.151:6789/0 says I am v1:172.21.15.28:42060/0 (socket says 172.21.15.28:42060) 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 -- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 msgr2=0x7fa6640ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 -- 172.21.15.28:0/978827420 >> v1:172.21.15.151:6789/0 conn(0x7fa6640cd0e0 legacy=0x7fa66415e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661e00640 1 -- 172.21.15.28:0/978827420 --> v2:172.21.15.151:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66415b4f0 con 0x7fa6640b43e0 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa661400640 1 -- 172.21.15.28:0/978827420 <== mon.1 v1:172.21.15.151:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773051809 0 0) 0x7fa66415bfd0 con 0x7fa6640cd0e0 2024-03-21T08:44:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.692+0000 7fa65be00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa661e00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640b43e0 0x7fa664071a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa654009870 tx=0x7fa65401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7915113ec7ebfda8 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa661400640 1 -- 172.21.15.28:0/978827420 <== mon.1 v2:172.21.15.151:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540b2070 con 0x7fa6640b43e0 2024-03-21T08:44:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa661400640 1 -- 172.21.15.28:0/978827420 <== mon.1 v2:172.21.15.151:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6540ad640 con 0x7fa6640b43e0 2024-03-21T08:44:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa661400640 1 -- 172.21.15.28:0/978827420 <== mon.1 v2:172.21.15.151:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540ada10 con 0x7fa6640b43e0 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640b43e0 msgr2=0x7fa664071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640b43e0 0x7fa664071a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa654009870 tx=0x7fa65401c560 comp rx=0 tx=0).stop 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- 172.21.15.28:0/978827420 shutdown_connections 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa6640ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 --2- 172.21.15.28:0/978827420 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640b43e0 0x7fa664071a20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- 172.21.15.28:0/978827420 >> 172.21.15.28:0/978827420 conn(0x7fa6640674f0 msgr2=0x7fa66406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- 172.21.15.28:0/978827420 shutdown_connections 2024-03-21T08:44:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- 172.21.15.28:0/978827420 wait complete. 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 Processor -- start 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.696+0000 7fa668e00640 1 -- start start 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa66414a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 0x7fa66406df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640cd0e0 0x7fa66406e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640cd4f0 con 0x7fa6640b43e0 2024-03-21T08:44:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b3a70 con 0x7fa6640cd0e0 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640cd670 con 0x7fa664071f60 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 0x7fa66406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa66414a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 0x7fa66406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33390/0 (socket says 172.21.15.28:33390) 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65be00640 1 -- 172.21.15.28:0/1135591895 learned_addr learned my addr 172.21.15.28:0/1135591895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa662800640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640cd0e0 0x7fa66406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 -- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640cd0e0 msgr2=0x7fa66406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fa6640cd0e0 0x7fa66406e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 -- 172.21.15.28:0/1135591895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 msgr2=0x7fa66406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 0x7fa66406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6641526e0 con 0x7fa664071f60 2024-03-21T08:44:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65be00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6640b43e0 0x7fa66406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-21T08:44:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa661e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa66414a4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa66406fee0 tx=0x7fa654019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540b2040 con 0x7fa664071f60 2024-03-21T08:44:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6540bb070 con 0x7fa664071f60 2024-03-21T08:44:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6641529c0 con 0x7fa664071f60 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540b66a0 con 0x7fa664071f60 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa664152f80 con 0x7fa664071f60 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.700+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa664068110 con 0x7fa664071f60 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7fa6540b6840 con 0x7fa664071f60 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa640067b20 0x7fa640069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa654120540 con 0x7fa664071f60 2024-03-21T08:44:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7fa64006dfc0 0x7fa640070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6540ad640 con 0x7fa64006dfc0 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== mon.2 v2:172.21.15.151:3301/0 6 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6541208f0 con 0x7fa664071f60 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65be00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa640067b20 0x7fa640069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa662800640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7fa64006dfc0 0x7fa640070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65be00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa640067b20 0x7fa640069fe0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa66414bf80 tx=0x7fa64c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa662800640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7fa64006dfc0 0x7fa640070400 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.704+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== osd.7 v2:172.21.15.151:6824/1764908544 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa6540ad640 con 0x7fa64006dfc0 2024-03-21T08:44:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 --> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa66406ec40 con 0x7fa64006dfc0 2024-03-21T08:44:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa65a000640 1 -- 172.21.15.28:0/1135591895 <== osd.7 v2:172.21.15.151:6824/1764908544 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa66406ec40 con 0x7fa64006dfc0 2024-03-21T08:44:24.721 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-21T08:44:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-21T08:44:24.721 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7fa64006dfc0 msgr2=0x7fa640070400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6824/1764908544,v1:172.21.15.151:6825/1764908544] conn(0x7fa64006dfc0 0x7fa640070400 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa640067b20 msgr2=0x7fa640069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7fa640067b20 0x7fa640069fe0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa66414bf80 tx=0x7fa64c009190 comp rx=0 tx=0).stop 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 msgr2=0x7fa66414a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 --2- 172.21.15.28:0/1135591895 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7fa664071f60 0x7fa66414a4a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa66406fee0 tx=0x7fa654019300 comp rx=0 tx=0).stop 2024-03-21T08:44:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa662800640 1 -- 172.21.15.28:0/1135591895 reap_dead start 2024-03-21T08:44:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 shutdown_connections 2024-03-21T08:44:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 >> 172.21.15.28:0/1135591895 conn(0x7fa6640674f0 msgr2=0x7fa6640b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 shutdown_connections 2024-03-21T08:44:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.720+0000 7fa668e00640 1 -- 172.21.15.28:0/1135591895 wait complete. 2024-03-21T08:44:24.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-21T08:44:24.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 Processor -- start 2024-03-21T08:44:24.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 -- start start 2024-03-21T08:44:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80f8144690 0x7f80f80b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 0x7f80f8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f80723e0 con 0x7f80f81442b0 2024-03-21T08:44:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f80b2ad0 con 0x7f80f8144690 2024-03-21T08:44:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80fe200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f80b2d20 con 0x7f80f8071a80 2024-03-21T08:44:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.880+0000 7f80eea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80f8144690 0x7f80f80b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80f8144690 0x7f80f80b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33396/0 (socket says 172.21.15.28:33396) 2024-03-21T08:44:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 -- 172.21.15.28:0/2440388140 learned_addr learned my addr 172.21.15.28:0/2440388140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80f7e00640 1 --2- 172.21.15.28:0/2440388140 >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 0x7f80f8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 -- 172.21.15.28:0/2440388140 >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 msgr2=0x7f80f8071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 --2- 172.21.15.28:0/2440388140 >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 0x7f80f8071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 -- 172.21.15.28:0/2440388140 >> v1:172.21.15.151:6789/0 conn(0x7f80f81442b0 legacy=0x7f80f80aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 -- 172.21.15.28:0/2440388140 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f80b3000 con 0x7f80f8144690 2024-03-21T08:44:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80f7e00640 1 --2- 172.21.15.28:0/2440388140 >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 0x7f80f8071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80eea00640 1 --2- 172.21.15.28:0/2440388140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f8144690 0x7f80f80b3dc0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f80f8072d60 tx=0x7f80e001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306368b35b50d352 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80f6a00640 1 -- 172.21.15.28:0/2440388140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00aa610 con 0x7f80f8144690 2024-03-21T08:44:24.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80f6a00640 1 -- 172.21.15.28:0/2440388140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e00af070 con 0x7f80f8144690 2024-03-21T08:44:24.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.884+0000 7f80f6a00640 1 -- 172.21.15.28:0/2440388140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00aa9d0 con 0x7f80f8144690 2024-03-21T08:44:24.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- 172.21.15.28:0/2440388140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f8144690 msgr2=0x7f80f80b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- 172.21.15.28:0/2440388140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f8144690 0x7f80f80b3dc0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f80f8072d60 tx=0x7f80e001c4b0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- 172.21.15.28:0/2440388140 shutdown_connections 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- 172.21.15.28:0/2440388140 >> v2:172.21.15.151:3300/0 conn(0x7f80f8071a80 0x7f80f8071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- 172.21.15.28:0/2440388140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f8144690 0x7f80f80b3dc0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- 172.21.15.28:0/2440388140 >> 172.21.15.28:0/2440388140 conn(0x7f80f80674f0 msgr2=0x7f80f806b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- 172.21.15.28:0/2440388140 shutdown_connections 2024-03-21T08:44:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- 172.21.15.28:0/2440388140 wait complete. 2024-03-21T08:44:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 Processor -- start 2024-03-21T08:44:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- start start 2024-03-21T08:44:24.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80f8071a80 0x7f80f815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f8144690 0x7f80f8142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80af470 con 0x7f80f81442b0 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80af2f0 con 0x7f80f8144690 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80fe200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f806e2b0 con 0x7f80f8071a80 2024-03-21T08:44:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33404/0 (socket says 172.21.15.28:33404) 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 -- 172.21.15.28:0/501544132 learned_addr learned my addr 172.21.15.28:0/501544132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80f7e00640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f8144690 0x7f80f8142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80f7400640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80f8071a80 0x7f80f815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 -- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80f8071a80 msgr2=0x7f80f815c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80f8071a80 0x7f80f815c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 -- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f8144690 msgr2=0x7f80f8142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f8144690 0x7f80f8142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 -- 172.21.15.28:0/501544132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f813c560 con 0x7f80f81442b0 2024-03-21T08:44:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80eea00640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f80e001ab60 tx=0x7f80e001acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.888+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00b3800 con 0x7f80f81442b0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e00b39c0 con 0x7f80f81442b0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00b3db0 con 0x7f80f81442b0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f813bda0 con 0x7f80f81442b0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f8144f60 con 0x7f80f81442b0 2024-03-21T08:44:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f8073430 con 0x7f80f81442b0 2024-03-21T08:44:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f80e00b2380 con 0x7f80f81442b0 2024-03-21T08:44:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80efe00640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 0x7f80d406a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f80e011f900 con 0x7f80f81442b0 2024-03-21T08:44:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.892+0000 7f80f7400640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 0x7f80d406a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.896+0000 7f80f7400640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 0x7f80d406a010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f80e8005530 tx=0x7f80e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:24.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:24.896+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f80e00a7070 con 0x7f80f81442b0 2024-03-21T08:44:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.056+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f80f8073430 con 0x7f80d4067b50 2024-03-21T08:44:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80efe00640 1 -- 172.21.15.28:0/501544132 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f80f8073430 con 0x7f80d4067b50 2024-03-21T08:44:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-03-21T08:44:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 msgr2=0x7f80d406a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 0x7f80d406a010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f80e8005530 tx=0x7f80e8009040 comp rx=0 tx=0).stop 2024-03-21T08:44:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 msgr2=0x7f80f81434e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f80e001ab60 tx=0x7f80e001acf0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 shutdown_connections 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f80f8144690 0x7f80f8142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f81442b0 0x7f80f81434e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f80d4067b50 0x7f80d406a010 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 --2- 172.21.15.28:0/501544132 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f80f8071a80 0x7f80f815c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 >> 172.21.15.28:0/501544132 conn(0x7f80f80674f0 msgr2=0x7f80f806e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 shutdown_connections 2024-03-21T08:44:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.060+0000 7f80fe200640 1 -- 172.21.15.28:0/501544132 wait complete. 2024-03-21T08:44:25.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:44:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 Processor -- start 2024-03-21T08:44:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 -- start start 2024-03-21T08:44:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cfa80 0x7f29400cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f29400b2e30 0x7f294006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29400cc880 0x7f294006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294015e1c0 con 0x7f29400cc880 2024-03-21T08:44:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29400d03a0 con 0x7f29400b2e30 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2944c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294006b310 con 0x7f29400cfa80 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29400cc880 0x7f294006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29400cc880 0x7f294006eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33410/0 (socket says 172.21.15.28:33410) 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 -- 172.21.15.28:0/2241430416 learned_addr learned my addr 172.21.15.28:0/2241430416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293f400640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cfa80 0x7f29400cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f2937e00640 1 --2- 172.21.15.28:0/2241430416 >> v2:172.21.15.151:3300/0 conn(0x7f29400b2e30 0x7f294006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 -- 172.21.15.28:0/2241430416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cfa80 msgr2=0x7f29400cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cfa80 0x7f29400cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 -- 172.21.15.28:0/2241430416 >> v2:172.21.15.151:3300/0 conn(0x7f29400b2e30 msgr2=0x7f294006e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 --2- 172.21.15.28:0/2241430416 >> v2:172.21.15.151:3300/0 conn(0x7f29400b2e30 0x7f294006e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.224+0000 7f293fe00640 1 -- 172.21.15.28:0/2241430416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294006b5f0 con 0x7f29400cc880 2024-03-21T08:44:25.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f293fe00640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cc880 0x7f294006eee0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2940072310 tx=0x7f293801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0279d5790faf9db server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:25.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f293ea00640 1 -- 172.21.15.28:0/2241430416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380a5860 con 0x7f29400cc880 2024-03-21T08:44:25.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f293ea00640 1 -- 172.21.15.28:0/2241430416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380a5a00 con 0x7f29400cc880 2024-03-21T08:44:25.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f293ea00640 1 -- 172.21.15.28:0/2241430416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380a5dd0 con 0x7f29400cc880 2024-03-21T08:44:25.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- 172.21.15.28:0/2241430416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cc880 msgr2=0x7f294006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cc880 0x7f294006eee0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2940072310 tx=0x7f293801c9c0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- 172.21.15.28:0/2241430416 shutdown_connections 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cc880 0x7f294006eee0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 --2- 172.21.15.28:0/2241430416 >> v2:172.21.15.151:3300/0 conn(0x7f29400b2e30 0x7f294006e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 --2- 172.21.15.28:0/2241430416 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cfa80 0x7f29400cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- 172.21.15.28:0/2241430416 >> 172.21.15.28:0/2241430416 conn(0x7f29400b2280 msgr2=0x7f29400b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- 172.21.15.28:0/2241430416 shutdown_connections 2024-03-21T08:44:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- 172.21.15.28:0/2241430416 wait complete. 2024-03-21T08:44:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 Processor -- start 2024-03-21T08:44:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.228+0000 7f2944c00640 1 -- start start 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 0x7f294014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 0x7f2940156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400cd770 con 0x7f29400cfa80 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294006f830 con 0x7f29400b2e30 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400cd8f0 con 0x7f29400cc880 2024-03-21T08:44:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33422/0 (socket says 172.21.15.28:33422) 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 -- 172.21.15.28:0/843126556 learned_addr learned my addr 172.21.15.28:0/843126556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 -- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 msgr2=0x7f2940156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937e00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 0x7f2940156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293f400640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 0x7f294014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 0x7f2940156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 -- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 msgr2=0x7f294014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 0x7f294014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 -- 172.21.15.28:0/843126556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29401525a0 con 0x7f29400cfa80 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293f400640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 0x7f294014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937e00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 0x7f2940156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f293fe00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f293801b070 tx=0x7f2938019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380b8670 con 0x7f29400cfa80 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380be070 con 0x7f29400cfa80 2024-03-21T08:44:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380b8a70 con 0x7f29400cfa80 2024-03-21T08:44:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294015ec20 con 0x7f29400cfa80 2024-03-21T08:44:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294014f670 con 0x7f29400cfa80 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2940075270 con 0x7f29400cfa80 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.232+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f29380b8c10 con 0x7f29400cfa80 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.236+0000 7f2937400640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 0x7f291c06a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.236+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2938120ae0 con 0x7f29400cfa80 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.236+0000 7f293f400640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 0x7f291c06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.236+0000 7f293f400640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 0x7f291c06a010 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f29300099c0 tx=0x7f2930008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:25.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.240+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f29380165e0 con 0x7f29400cfa80 2024-03-21T08:44:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.400+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29401552d0 con 0x7f291c067b50 2024-03-21T08:44:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2937400640 1 -- 172.21.15.28:0/843126556 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13469 (secure 0 0 0) 0x7f29401552d0 con 0x7f291c067b50 2024-03-21T08:44:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 msgr2=0x7f291c06a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 0x7f291c06a010 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f29300099c0 tx=0x7f2930008040 comp rx=0 tx=0).stop 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 msgr2=0x7f2940151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f293801b070 tx=0x7f2938019760 comp rx=0 tx=0).stop 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 shutdown_connections 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29400cfa80 0x7f2940151aa0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f291c067b50 0x7f291c06a010 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f29400cc880 0x7f2940156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 --2- 172.21.15.28:0/843126556 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f29400b2e30 0x7f294014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 >> 172.21.15.28:0/843126556 conn(0x7f29400b2280 msgr2=0x7f2940158520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 shutdown_connections 2024-03-21T08:44:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:25.404+0000 7f2944c00640 1 -- 172.21.15.28:0/843126556 wait complete. 2024-03-21T08:44:25.424 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-03-21T08:44:23.765840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239320,"kb_used_data":6504,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735400,"statfs":{"total":773094113280,"available":772849049600,"internally_reserved":0,"allocated":6660096,"data_stored":3634544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.561341"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":237,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-21T08:44:16.561744+0000","last_change":"2024-03-21T08:40:55.710171+0000","last_active":"2024-03-21T08:44:16.561744+0000","last_peered":"2024-03-21T08:44:16.561744+0000","last_clean":"2024-03-21T08:44:16.561744+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:44:16.561744+0000","last_undegraded":"2024-03-21T08:44:16.561744+0000","last_fullsized":"2024-03-21T08:44:16.561744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T08:40:53.889834+0000","last_clean_scrub_stamp":"2024-03-21T08:40:53.889834+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T10:07:37.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509485,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1277952,"data_stored":897094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509484,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542188,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542188,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1277952,"data_stored":897094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:44:25.426 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-21T08:40:53.889834+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=21, tm_hour=8, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=81, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=21, tm_hour=8, tm_min=44, tm_sec=19, tm_wday=3, tm_yday=81, tm_isdst=0) 2024-03-21T08:44:25.426 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-21T08:44:38.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:38.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:45.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-21T08:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 Processor -- start 2024-03-21T08:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 -- start start 2024-03-21T08:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 0x7f721c15e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f721c15ba70 0x7f721c0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 0x7f721c071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c15bf00 con 0x7f721c15ba70 2024-03-21T08:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c0cedb0 con 0x7f721c0ce490 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7223000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c15ebf0 con 0x7f721c071980 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f721c15ba70 0x7f721c0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f721c15ba70 0x7f721c0ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42478/0 (socket says 172.21.15.28:42478) 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 -- 172.21.15.28:0/2828724711 learned_addr learned my addr 172.21.15.28:0/2828724711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7221000640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 0x7f721c071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f721be00640 1 --2- 172.21.15.28:0/2828724711 >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 0x7f721c15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 -- 172.21.15.28:0/2828724711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 msgr2=0x7f721c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 0x7f721c071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 -- 172.21.15.28:0/2828724711 >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 msgr2=0x7f721c15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:45.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 --2- 172.21.15.28:0/2828724711 >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 0x7f721c15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7213e00640 1 -- 172.21.15.28:0/2828724711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c15eed0 con 0x7f721c15ba70 2024-03-21T08:44:45.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f7221000640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 0x7f721c071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:45.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.580+0000 7f721be00640 1 --2- 172.21.15.28:0/2828724711 >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 0x7f721c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7213e00640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c15ba70 0x7f721c0ce870 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f721c0730c0 tx=0x7f720401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9817c89a22c0d7d server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f721b400640 1 -- 172.21.15.28:0/2828724711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72040b0070 con 0x7f721c15ba70 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f721b400640 1 -- 172.21.15.28:0/2828724711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72040ab600 con 0x7f721c15ba70 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f721b400640 1 -- 172.21.15.28:0/2828724711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72040ab9d0 con 0x7f721c15ba70 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 -- 172.21.15.28:0/2828724711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c15ba70 msgr2=0x7f721c0ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c15ba70 0x7f721c0ce870 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f721c0730c0 tx=0x7f720401c490 comp rx=0 tx=0).stop 2024-03-21T08:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 -- 172.21.15.28:0/2828724711 shutdown_connections 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c071980 0x7f721c071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 --2- 172.21.15.28:0/2828724711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c15ba70 0x7f721c0ce870 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 --2- 172.21.15.28:0/2828724711 >> v2:172.21.15.151:3300/0 conn(0x7f721c0ce490 0x7f721c15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 -- 172.21.15.28:0/2828724711 >> 172.21.15.28:0/2828724711 conn(0x7f721c0cc230 msgr2=0x7f721c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 -- 172.21.15.28:0/2828724711 shutdown_connections 2024-03-21T08:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.584+0000 7f7223000640 1 -- 172.21.15.28:0/2828724711 wait complete. 2024-03-21T08:44:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 Processor -- start 2024-03-21T08:44:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- start start 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f721c071980 0x7f721c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 0x7f721c164880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0726d0 con 0x7f721c0ce490 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c155390 con 0x7f721c071980 2024-03-21T08:44:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0b4b90 con 0x7f721c15ba70 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42482/0 (socket says 172.21.15.28:42482) 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 -- 172.21.15.28:0/62430050 learned_addr learned my addr 172.21.15.28:0/62430050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f721be00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f721c071980 0x7f721c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7221000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 0x7f721c164880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 -- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 msgr2=0x7f721c164880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 0x7f721c164880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 -- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f721c071980 msgr2=0x7f721c148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f721c071980 0x7f721c148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 -- 172.21.15.28:0/62430050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c14acf0 con 0x7f721c0ce490 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7221000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 0x7f721c164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7213e00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f720401cd30 tx=0x7f7204019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72040b9070 con 0x7f721c0ce490 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72040b4640 con 0x7f721c0ce490 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c14afd0 con 0x7f721c0ce490 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72040b0040 con 0x7f721c0ce490 2024-03-21T08:44:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.588+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c164360 con 0x7f721c0ce490 2024-03-21T08:44:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c04f410 con 0x7f721c0ce490 2024-03-21T08:44:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f72040bc070 con 0x7f721c0ce490 2024-03-21T08:44:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f7218c00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 0x7f71f8069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f720411ef90 con 0x7f721c0ce490 2024-03-21T08:44:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f721be00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 0x7f71f8069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.592+0000 7f721be00640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 0x7f71f8069fe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f720c00a9a0 tx=0x7f720c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.596+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7204016110 con 0x7f721c0ce490 2024-03-21T08:44:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.756+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 --> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f721c072c10 con 0x7f71f8067b20 2024-03-21T08:44:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7218c00640 1 -- 172.21.15.28:0/62430050 <== mgr.4105 v2:172.21.15.151:6832/2277034983 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13359 (secure 0 0 0) 0x7f721c072c10 con 0x7f71f8067b20 2024-03-21T08:44:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-21T08:44:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 msgr2=0x7f71f8069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 0x7f71f8069fe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f720c00a9a0 tx=0x7f720c008040 comp rx=0 tx=0).stop 2024-03-21T08:44:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 msgr2=0x7f721c14a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f720401cd30 tx=0x7f7204019230 comp rx=0 tx=0).stop 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 shutdown_connections 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f71f8067b20 0x7f71f8069fe0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f721c15ba70 0x7f721c164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f721c0ce490 0x7f721c14a2a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 --2- 172.21.15.28:0/62430050 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f721c071980 0x7f721c148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 >> 172.21.15.28:0/62430050 conn(0x7f721c0cc230 msgr2=0x7f721c06d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 shutdown_connections 2024-03-21T08:44:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.760+0000 7f7223000640 1 -- 172.21.15.28:0/62430050 wait complete. 2024-03-21T08:44:45.780 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-03-21T08:44:43.770325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239308,"kb_used_data":6504,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735412,"statfs":{"total":773094113280,"available":772849061888,"internally_reserved":0,"allocated":6660096,"data_stored":3634544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"1.0","version":"57'119","reported_seq":248,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-21T08:44:25.143480+0000","last_change":"2024-03-21T08:44:25.143519+0000","last_active":"2024-03-21T08:44:25.143480+0000","last_peered":"2024-03-21T08:44:25.143480+0000","last_clean":"2024-03-21T08:44:25.143480+0000","last_became_active":"2024-03-21T08:40:55.708450+0000","last_became_peered":"2024-03-21T08:40:55.708450+0000","last_unstale":"2024-03-21T08:44:25.143480+0000","last_undegraded":"2024-03-21T08:44:25.143480+0000","last_fullsized":"2024-03-21T08:44:25.143480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'119","last_scrub_stamp":"2024-03-21T08:44:25.143378+0000","last_deep_scrub":"57'119","last_deep_scrub_stamp":"2024-03-21T08:44:25.143378+0000","last_clean_scrub_stamp":"2024-03-21T08:44:25.143378+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T13:21:34.274714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509489,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1277952,"data_stored":897094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509488,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542192,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":306726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542192,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1277952,"data_stored":897094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T08:44:45.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-21T08:44:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 Processor -- start 2024-03-21T08:44:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 -- start start 2024-03-21T08:44:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 0x7f5cd80cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cd806b170 0x7f5cd806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd80d0760 con 0x7f5cd815e020 2024-03-21T08:44:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd80ce7b0 con 0x7f5cd806b170 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdea00640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd80cfc90 con 0x7f5cd80ccb70 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cd806b170 0x7f5cd806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cd806b170 0x7f5cd806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42484/0 (socket says 172.21.15.28:42484) 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cdca00640 1 -- 172.21.15.28:0/2580245975 learned_addr learned my addr 172.21.15.28:0/2580245975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.920+0000 7f5cd7400640 1 --2- 172.21.15.28:0/2580245975 >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 0x7f5cd80cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdca00640 1 -- 172.21.15.28:0/2580245975 >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 msgr2=0x7f5cd80cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdca00640 1 --2- 172.21.15.28:0/2580245975 >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 0x7f5cd80cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdca00640 1 -- 172.21.15.28:0/2580245975 >> v1:172.21.15.151:6789/0 conn(0x7f5cd815e020 legacy=0x7f5cd80cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdca00640 1 -- 172.21.15.28:0/2580245975 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80cfec0 con 0x7f5cd806b170 2024-03-21T08:44:45.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cd7400640 1 --2- 172.21.15.28:0/2580245975 >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 0x7f5cd80cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:45.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdca00640 1 --2- 172.21.15.28:0/2580245975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd806b170 0x7f5cd806b5c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009e50 tx=0x7f5cd001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6264c8ca040bdbb server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cd6a00640 1 -- 172.21.15.28:0/2580245975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd00a56c0 con 0x7f5cd806b170 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cd6a00640 1 -- 172.21.15.28:0/2580245975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd00a5860 con 0x7f5cd806b170 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cd6a00640 1 -- 172.21.15.28:0/2580245975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd00a5c30 con 0x7f5cd806b170 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdea00640 1 -- 172.21.15.28:0/2580245975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd806b170 msgr2=0x7f5cd806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.924+0000 7f5cdea00640 1 --2- 172.21.15.28:0/2580245975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd806b170 0x7f5cd806b5c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009e50 tx=0x7f5cd001c9f0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- 172.21.15.28:0/2580245975 shutdown_connections 2024-03-21T08:44:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 --2- 172.21.15.28:0/2580245975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd806b170 0x7f5cd806b5c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 --2- 172.21.15.28:0/2580245975 >> v2:172.21.15.151:3300/0 conn(0x7f5cd80ccb70 0x7f5cd80cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- 172.21.15.28:0/2580245975 >> 172.21.15.28:0/2580245975 conn(0x7f5cd8099a30 msgr2=0x7f5cd8099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- 172.21.15.28:0/2580245975 shutdown_connections 2024-03-21T08:44:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- 172.21.15.28:0/2580245975 wait complete. 2024-03-21T08:44:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 Processor -- start 2024-03-21T08:44:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- start start 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 0x7f5cd8145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cd815e020 0x7f5cd8159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815f410 con 0x7f5cd80ccb70 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815e790 con 0x7f5cd815e020 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815f590 con 0x7f5cd806b170 2024-03-21T08:44:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42486/0 (socket says 172.21.15.28:42486) 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 -- 172.21.15.28:0/1511480896 learned_addr learned my addr 172.21.15.28:0/1511480896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdca00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cd815e020 0x7f5cd8159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7e00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 0x7f5cd8145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 -- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 msgr2=0x7f5cd8145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 0x7f5cd8145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 -- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cd815e020 msgr2=0x7f5cd8159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cd815e020 0x7f5cd8159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 -- 172.21.15.28:0/1511480896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd80673c0 con 0x7f5cd80ccb70 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7e00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 0x7f5cd8145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cd7400640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e840 tx=0x7f5cc400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4017070 con 0x7f5cd80ccb70 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc400c8d0 con 0x7f5cd80ccb70 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400f040 con 0x7f5cd80ccb70 2024-03-21T08:44:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.928+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80c7350 con 0x7f5cd80ccb70 2024-03-21T08:44:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80c7070 con 0x7f5cd80ccb70 2024-03-21T08:44:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd809eb60 con 0x7f5cd80ccb70 2024-03-21T08:44:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f5cc401a070 con 0x7f5cd80ccb70 2024-03-21T08:44:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 0x7f5cb806a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5cc40859a0 con 0x7f5cd80ccb70 2024-03-21T08:44:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5cd7e00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 0x7f5cb806a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.932+0000 7f5cd7e00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 0x7f5cb806a010 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8001710 tx=0x7f5cc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:45.936+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5cc4055da0 con 0x7f5cd80ccb70 2024-03-21T08:44:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.100+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f5cd804f4f0 con 0x7f5cd80ccb70 2024-03-21T08:44:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.104+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) v1 ==== 123+0+0 (secure 0 0 0) 0x7f5cc4059e40 con 0x7f5cd80ccb70 2024-03-21T08:44:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.104+0000 7f5ccfe00640 1 -- 172.21.15.28:0/1511480896 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5cc408b080 con 0x7f5cd80ccb70 2024-03-21T08:44:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 msgr2=0x7f5cb806a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 0x7f5cb806a010 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8001710 tx=0x7f5cc8009040 comp rx=0 tx=0).stop 2024-03-21T08:44:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 msgr2=0x7f5cd8159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400e840 tx=0x7f5cc400ed00 comp rx=0 tx=0).stop 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 shutdown_connections 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f5cd815e020 0x7f5cd8159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cd80ccb70 0x7f5cd8159020 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f5cb8067b50 0x7f5cb806a010 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 --2- 172.21.15.28:0/1511480896 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f5cd806b170 0x7f5cd8145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 >> 172.21.15.28:0/1511480896 conn(0x7f5cd8099a30 msgr2=0x7f5cd815bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 shutdown_connections 2024-03-21T08:44:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.108+0000 7f5cdea00640 1 -- 172.21.15.28:0/1511480896 wait complete. 2024-03-21T08:44:46.127 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-21T08:44:46.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-21T08:44:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.268+0000 7f1980200640 1 Processor -- start 2024-03-21T08:44:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.268+0000 7f1980200640 1 -- start start 2024-03-21T08:44:46.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1980200640 1 --2- >> v2:172.21.15.151:3300/0 conn(0x7f197815ba70 0x7f19780b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1980200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197815ebf0 con 0x7f197815e610 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1980200640 1 -- --> v1:172.21.15.151:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197815eed0 con 0x7f19780ce490 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1980200640 1 -- --> v2:172.21.15.151:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197815f1b0 con 0x7f197815ba70 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197ce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f197815e610 0x7f19780b4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37910/0 (socket says 172.21.15.28:37910) 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197ce00640 1 -- 172.21.15.28:0/1795301720 learned_addr learned my addr 172.21.15.28:0/1795301720 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 --2- 172.21.15.28:0/1795301720 >> v2:172.21.15.151:3300/0 conn(0x7f197815ba70 0x7f19780b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554914844 0 0) 0x7f197815ebf0 con 0x7f197815e610 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1960003530 con 0x7f197815e610 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702261384 0 0) 0x7f1960003530 con 0x7f197815e610 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 >> v2:172.21.15.151:3300/0 conn(0x7f197815ba70 msgr2=0x7f19780b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 --2- 172.21.15.28:0/1795301720 >> v2:172.21.15.151:3300/0 conn(0x7f197815ba70 0x7f19780b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 >> v1:172.21.15.151:6789/0 conn(0x7f19780ce490 legacy=0x7f19780ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197815f490 con 0x7f197815e610 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1762784645 0 0) 0x7f1964001680 con 0x7f197815e610 2024-03-21T08:44:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 >> v1:172.21.15.28:6789/0 conn(0x7f197815e610 legacy=0x7f19780b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1960003a10 0x7f1960003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1960004810 0x7f1960004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f196000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197815f1b0 con 0x7f1960005220 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196000c170 con 0x7f1960004810 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196000c370 con 0x7f1960003a10 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f196000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197d800640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1960003a10 0x7f1960003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197ce00640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1960004810 0x7f1960004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 -- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1960003a10 msgr2=0x7f1960003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f1960003a10 0x7f1960003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 -- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1960004810 msgr2=0x7f1960004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1960004810 0x7f1960004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.272+0000 7f197e200640 1 -- 172.21.15.28:0/1795301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960003530 con 0x7f1960005220 2024-03-21T08:44:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f197e200640 1 -- 172.21.15.28:0/1795301720 reap_dead start 2024-03-21T08:44:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f197e200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f196000b9d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1970009f90 tx=0x7f197001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb0903a22db26c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970009c10 con 0x7f1960005220 2024-03-21T08:44:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f19700a5c90 con 0x7f1960005220 2024-03-21T08:44:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1977e00640 1 -- 172.21.15.28:0/1795301720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964007e60 con 0x7f1960005220 2024-03-21T08:44:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 -- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 msgr2=0x7f196000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f196000b9d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1970009f90 tx=0x7f197001cc80 comp rx=0 tx=0).stop 2024-03-21T08:44:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 -- 172.21.15.28:0/1795301720 shutdown_connections 2024-03-21T08:44:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 --2- 172.21.15.28:0/1795301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f196000b9d0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 -- 172.21.15.28:0/1795301720 >> 172.21.15.28:0/1795301720 conn(0x7f19780cc230 msgr2=0x7f19780cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 -- 172.21.15.28:0/1795301720 shutdown_connections 2024-03-21T08:44:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 -- 172.21.15.28:0/1795301720 wait complete. 2024-03-21T08:44:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.276+0000 7f1980200640 1 Processor -- start 2024-03-21T08:44:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- start start 2024-03-21T08:44:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 --2- >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 0x7f1978153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 --2- >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 0x7f197815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1978155390 con 0x7f1960005220 2024-03-21T08:44:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- --> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197815bea0 con 0x7f1978146ad0 2024-03-21T08:44:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- --> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19780ce870 con 0x7f197815d1a0 2024-03-21T08:44:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42508/0 (socket says 172.21.15.28:42508) 2024-03-21T08:44:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 -- 172.21.15.28:0/2767376439 learned_addr learned my addr 172.21.15.28:0/2767376439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-21T08:44:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197e200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 0x7f197815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197ce00640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 0x7f1978153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 -- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 msgr2=0x7f197815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 0x7f197815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 -- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 msgr2=0x7f1978153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 0x7f1978153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 -- 172.21.15.28:0/2767376439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1978147000 con 0x7f1960005220 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197e200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 0x7f197815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197ce00640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 0x7f1978153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-21T08:44:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f197d800640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f196c00f540 tx=0x7f196c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c0052e0 con 0x7f1960005220 2024-03-21T08:44:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f196c018070 con 0x7f1960005220 2024-03-21T08:44:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19781472e0 con 0x7f1960005220 2024-03-21T08:44:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c005480 con 0x7f1960005220 2024-03-21T08:44:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19781478a0 con 0x7f1960005220 2024-03-21T08:44:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.280+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197804f410 con 0x7f1960005220 2024-03-21T08:44:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.284+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 87459+0+0 (secure 0 0 0) 0x7f196c003820 con 0x7f1960005220 2024-03-21T08:44:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.284+0000 7f1975600640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 0x7f1958069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-21T08:44:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.284+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f196c085c00 con 0x7f1960005220 2024-03-21T08:44:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.284+0000 7f197ce00640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 0x7f1958069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-21T08:44:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.284+0000 7f197ce00640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 0x7f1958069fe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1964000c10 tx=0x7f1964001de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-21T08:44:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.288+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f196c056650 con 0x7f1960005220 2024-03-21T08:44:46.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1978072c10 con 0x7f1960005220 2024-03-21T08:44:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1975600640 1 -- 172.21.15.28:0/2767376439 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f196c055f50 con 0x7f1960005220 2024-03-21T08:44:46.483 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-21T08:44:46.484 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5591,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6836/3616703745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6834","nonce":3616703745},{"type":"v1","addr":"172.21.15.151:6836","nonce":3616703745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":6124,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.151:6837/1222996822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6835","nonce":1222996822},{"type":"v1","addr":"172.21.15.151:6837","nonce":1222996822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6243,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/595318418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":595318418},{"type":"v1","addr":"172.21.15.28:6833","nonce":595318418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1964916533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1964916533},{"type":"v1","addr":"172.21.15.28:6835","nonce":1964916533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-03-21T08:44:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-03-21T08:44:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 msgr2=0x7f1958069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 0x7f1958069fe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1964000c10 tx=0x7f1964001de0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 msgr2=0x7f19781441a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-21T08:44:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f196c00f540 tx=0x7f196c00fa00 comp rx=0 tx=0).stop 2024-03-21T08:44:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 shutdown_connections 2024-03-21T08:44:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:6832/2277034983,v1:172.21.15.151:6833/2277034983] conn(0x7f1958067b20 0x7f1958069fe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3301/0,v1:172.21.15.151:6790/0] conn(0x7f197815d1a0 0x7f197815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f1978146ad0 0x7f1978153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 --2- 172.21.15.28:0/2767376439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1960005220 0x7f19781441a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-21T08:44:46.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 >> 172.21.15.28:0/2767376439 conn(0x7f19780cc230 msgr2=0x7f19780cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-21T08:44:46.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 shutdown_connections 2024-03-21T08:44:46.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-21T08:44:46.484+0000 7f1980200640 1 -- 172.21.15.28:0/2767376439 wait complete. 2024-03-21T08:44:46.505 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-21T08:44:46.505 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-21T08:44:46.505 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-21T08:44:46.505 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.517 INFO:tasks.ceph.mds.a:Stopped 2024-03-21T08:44:46.517 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-21T08:44:46.518 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.575 INFO:tasks.ceph.mds.c:Stopped 2024-03-21T08:44:46.575 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-21T08:44:46.575 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.590 INFO:tasks.ceph.mds.b:Stopped 2024-03-21T08:44:46.590 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-21T08:44:46.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.646 INFO:tasks.ceph.mds.d:Stopped 2024-03-21T08:44:46.647 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-21T08:44:46.647 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-21T08:44:46.647 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.722 INFO:tasks.ceph.osd.0:Stopped 2024-03-21T08:44:46.722 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-21T08:44:46.723 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.800 INFO:tasks.ceph.osd.1:Stopped 2024-03-21T08:44:46.800 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-21T08:44:46.800 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:46.892 INFO:tasks.ceph.osd.2:Stopped 2024-03-21T08:44:46.892 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-21T08:44:46.892 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:49.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-21T08:44:52.995 INFO:tasks.ceph.osd.3:Stopped 2024-03-21T08:44:52.996 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-21T08:44:52.996 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.070 INFO:tasks.ceph.osd.4:Stopped 2024-03-21T08:44:53.071 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-21T08:44:53.071 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.157 INFO:tasks.ceph.osd.5:Stopped 2024-03-21T08:44:53.158 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-21T08:44:53.158 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.241 INFO:tasks.ceph.osd.6:Stopped 2024-03-21T08:44:53.241 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-21T08:44:53.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.322 INFO:tasks.ceph.osd.7:Stopped 2024-03-21T08:44:53.323 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-21T08:44:53.323 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-21T08:44:53.323 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.378 INFO:tasks.ceph.mgr.y:Stopped 2024-03-21T08:44:53.378 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-21T08:44:53.378 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.436 INFO:tasks.ceph.mgr.x:Stopped 2024-03-21T08:44:53.436 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-21T08:44:53.436 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-21T08:44:53.437 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.456 INFO:tasks.ceph.mon.a:Stopped 2024-03-21T08:44:53.457 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-21T08:44:53.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.473 INFO:tasks.ceph.mon.b:Stopped 2024-03-21T08:44:53.473 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-21T08:44:53.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-21T08:44:53.532 INFO:tasks.ceph.mon.c:Stopped 2024-03-21T08:44:53.533 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-21T08:44:53.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-03-21T08:44:53.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:53.569 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-21T08:44:53.843 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:53.843 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-21T08:44:54.068 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:54.069 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-21T08:44:54.285 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:54.285 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-21T08:44:54.459 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:54.459 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-21T08:44:54.700 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:54.700 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-21T08:44:54.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-21T08:44:54.897 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:54.897 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-21T08:44:55.038 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:55.038 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-21T08:44:55.187 INFO:tasks.ceph:Archiving mon data... 2024-03-21T08:44:55.189 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/data/mon.a.tgz 2024-03-21T08:44:55.189 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-03-21T08:44:55.199 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.8F1Q7jYdF4 2024-03-21T08:44:55.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8F1Q7jYdF4 2024-03-21T08:44:55.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.8F1Q7jYdF4 2024-03-21T08:44:55.415 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.8F1Q7jYdF4 is 365KB 2024-03-21T08:44:55.480 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.8F1Q7jYdF4 2024-03-21T08:44:55.489 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/data/mon.b.tgz 2024-03-21T08:44:55.489 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-21T08:44:55.498 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.mjY7yBCY3Y 2024-03-21T08:44:55.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.mjY7yBCY3Y 2024-03-21T08:44:55.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.mjY7yBCY3Y 2024-03-21T08:44:55.736 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.mjY7yBCY3Y is 482KB 2024-03-21T08:44:55.803 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.mjY7yBCY3Y 2024-03-21T08:44:55.811 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/data/mon.c.tgz 2024-03-21T08:44:55.812 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-21T08:44:55.860 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.vjNrGoq81J 2024-03-21T08:44:55.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.vjNrGoq81J 2024-03-21T08:44:56.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.vjNrGoq81J 2024-03-21T08:44:56.085 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.vjNrGoq81J is 482KB 2024-03-21T08:44:56.147 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.vjNrGoq81J 2024-03-21T08:44:56.155 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-21T08:44:56.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-21T08:44:56.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-21T08:44:56.231 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-21T08:44:56.264 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-21T08:44:56.265 INFO:tasks.ceph:Archiving crash dumps... 2024-03-21T08:44:56.267 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi028/crash 2024-03-21T08:44:56.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-21T08:44:56.294 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi151/crash 2024-03-21T08:44:56.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-21T08:44:56.326 INFO:tasks.ceph:Shutting down logrotate 2024-03-21T08:44:56.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:56.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:56.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:56.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-21T08:44:56.488 INFO:tasks.ceph:Compressing logs... 2024-03-21T08:44:56.488 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T08:44:56.512 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T08:44:56.540 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32391.log 2024-03-21T08:44:56.541 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39009.log 2024-03-21T08:44:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2024-03-21T08:44:56.542 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32391.log.gz 2024-03-21T08:44:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36942.log 2024-03-21T08:44:56.543 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39009.log.gz 2024-03-21T08:44:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33514.log 2024-03-21T08:44:56.544 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2024-03-21T08:44:56.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40723.log 2024-03-21T08:44:56.544 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36942.log.gz 2024-03-21T08:44:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41285.log 2024-03-21T08:44:56.545 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33514.log.gz 2024-03-21T08:44:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40062.log 2024-03-21T08:44:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34740.log 2024-03-21T08:44:56.547 INFO:teuthology.orchestra.run.smithi028.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.0.40723.log.gz 2024-03-21T08:44:56.547 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41285.log.gz 2024-03-21T08:44:56.547 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-03-21T08:44:56.548 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40062.log.gz 2024-03-21T08:44:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32365.log 2024-03-21T08:44:56.549 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34740.log.gz 2024-03-21T08:44:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2024-03-21T08:44:56.549 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-03-21T08:44:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-03-21T08:44:56.550 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32365.log.gz 2024-03-21T08:44:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-03-21T08:44:56.551 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2024-03-21T08:44:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36812.log 2024-03-21T08:44:56.552 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-03-21T08:44:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-03-21T08:44:56.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-03-21T08:44:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32682.log 2024-03-21T08:44:56.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36812.log.gz 2024-03-21T08:44:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41904.log 2024-03-21T08:44:56.554 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-03-21T08:44:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33800.log 2024-03-21T08:44:56.555 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32682.log.gz 2024-03-21T08:44:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2024-03-21T08:44:56.556 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41904.log.gz 2024-03-21T08:44:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34402.log 2024-03-21T08:44:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33800.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33800.log.gz 2024-03-21T08:44:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36996.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-21T08:44:56.557 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2024-03-21T08:44:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 2024-03-21T08:44:56.558 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34402.log.gz 2024-03-21T08:44:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32474.log 2024-03-21T08:44:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2024-03-21T08:44:56.564 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32474.log.gz 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2024-03-21T08:44:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42060.log: 93.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-21T08:44:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31848.log 2024-03-21T08:44:56.566 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-03-21T08:44:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31874.log 2024-03-21T08:44:56.566 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-03-21T08:44:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36732.log 2024-03-21T08:44:56.568 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31848.log.gz 2024-03-21T08:44:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32734.log 2024-03-21T08:44:56.568 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31874.log.gz 2024-03-21T08:44:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32788.log 2024-03-21T08:44:56.569 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36732.log.gz 2024-03-21T08:44:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-03-21T08:44:56.569 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32734.log.gz 2024-03-21T08:44:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32992.log 2024-03-21T08:44:56.569 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32788.log.gz 2024-03-21T08:44:56.575 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.32726.log 2024-03-21T08:44:56.575 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.32832.log 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.1.32726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.3.32832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.1.32726.log.gz 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27527.log: 91.5% -- replaced with /var/log/ceph/ceph-client.3.32832.log.gz 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27503.log 2024-03-21T08:44:56.576 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27551.log 2024-03-21T08:44:56.577 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27503.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-21T08:44:56.577 INFO:teuthology.orchestra.run.smithi151.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27503.log.gz 2024-03-21T08:44:56.577 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2024-03-21T08:44:56.577 INFO:teuthology.orchestra.run.smithi151.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27551.log.gz 2024-03-21T08:44:56.577 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27599.log 2024-03-21T08:44:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32992.log.gz 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31822.log 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-03-21T08:44:56.580 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31634.log 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31822.log.gz 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36376.log 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31634.log.gz 2024-03-21T08:44:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36376.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36376.log.gz 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38624.log 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31662.log 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38624.log.gz 2024-03-21T08:44:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41175.log 2024-03-21T08:44:56.583 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31662.log.gz 2024-03-21T08:44:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2024-03-21T08:44:56.583 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-03-21T08:44:56.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-03-21T08:44:56.583 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41175.log.gz 2024-03-21T08:44:56.588 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-21T08:44:56.588 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27623.log: /var/log/ceph/ceph-client.admin.27599.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2024-03-21T08:44:56.588 INFO:teuthology.orchestra.run.smithi151.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27599.log.gz 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27479.log 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27479.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27479.log.gz 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27647.log 2024-03-21T08:44:56.589 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-21T08:44:56.590 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.27647.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27647.log.gz 2024-03-21T08:44:56.593 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-21T08:44:56.593 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-21T08:44:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mds.b.log: 91.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-21T08:44:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-21T08:44:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-21T08:44:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27575.log 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38277.log 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32630.log 2024-03-21T08:44:56.597 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38277.log.gz 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35243.log 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38887.log 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32630.log.gz 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34350.log 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35243.log.gz 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38408.log 2024-03-21T08:44:56.598 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38887.log.gz 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34350.log.gz 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40192.log 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38408.log.gz 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35747.log 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37693.log 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40192.log.gz 2024-03-21T08:44:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40140.log 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31294.log 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37693.log.gz 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40140.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22492.log 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40140.log.gz 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37111.log 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31294.log.gz 2024-03-21T08:44:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.tmp-client.admin.22492.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22492.log.gz 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35034.log 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37111.log: gzip -5 88.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.37111.log.gz -- 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.39930.log 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35034.log.gz 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.34144.log 2024-03-21T08:44:56.601 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-03-21T08:44:56.603 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.27575.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27575.log.gz 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32526.log 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.2.34144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27521.log 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.2.34144.log.gz 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32526.log.gz 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35986.log 2024-03-21T08:44:56.607 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27521.log.gz 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35165.log 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35986.log.gz 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31606.log 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35165.log.gz 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31926.log 2024-03-21T08:44:56.608 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35139.log 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31606.log.gz 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35825.log 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.31926.log.gz 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37024.log 2024-03-21T08:44:56.609 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35139.log.gz 2024-03-21T08:44:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36246.log 2024-03-21T08:44:56.611 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35825.log.gz 2024-03-21T08:44:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37024.log.gz 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35560.log 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36246.log.gz 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38304.log 2024-03-21T08:44:56.612 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35560.log.gz 2024-03-21T08:44:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40008.log 2024-03-21T08:44:56.614 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-03-21T08:44:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38304.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-21T08:44:56.614 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38304.log.gz 2024-03-21T08:44:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34532.log 2024-03-21T08:44:56.614 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40008.log.gz 2024-03-21T08:44:56.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42268.log 2024-03-21T08:44:56.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34636.log 2024-03-21T08:44:56.616 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34532.log.gz 2024-03-21T08:44:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-03-21T08:44:56.616 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42268.log.gz 2024-03-21T08:44:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34636.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34636.log.gz 2024-03-21T08:44:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35430.log 2024-03-21T08:44:56.618 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33199.log 2024-03-21T08:44:56.618 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-03-21T08:44:56.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42320.log 2024-03-21T08:44:56.619 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35430.log.gz 2024-03-21T08:44:56.620 INFO:teuthology.orchestra.run.smithi151.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-21T08:44:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39757.log 2024-03-21T08:44:56.621 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33199.log.gz 2024-03-21T08:44:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-03-21T08:44:56.622 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42320.log.gz 2024-03-21T08:44:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39757.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-21T08:44:56.623 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39757.log.gz 2024-03-21T08:44:56.623 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39677.log 2024-03-21T08:44:56.623 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-03-21T08:44:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-03-21T08:44:56.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31900.log 2024-03-21T08:44:56.625 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39677.log.gz 2024-03-21T08:44:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-03-21T08:44:56.626 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2024-03-21T08:44:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31900.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.31900.log.gz 2024-03-21T08:44:56.628 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-03-21T08:44:56.628 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-03-21T08:44:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41982.log: /var/log/ceph/ceph-client.admin.39731.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-03-21T08:44:56.629 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-03-21T08:44:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35588.log 2024-03-21T08:44:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.40373.log: --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-03-21T08:44:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35588.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-03-21T08:44:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34246.log 2024-03-21T08:44:56.632 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35588.log.gz 2024-03-21T08:44:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-03-21T08:44:56.633 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-03-21T08:44:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35692.log 2024-03-21T08:44:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34246.log: /var/log/ceph/ceph-client.admin.31528.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34246.log.gz 2024-03-21T08:44:56.635 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz 2024-03-21T08:44:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-03-21T08:44:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35692.log: /var/log/ceph/ceph-client.admin.37245.log 2024-03-21T08:44:56.636 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35692.log.gz 2024-03-21T08:44:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2024-03-21T08:44:56.637 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-03-21T08:44:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34188.log 2024-03-21T08:44:56.638 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37245.log.gz 2024-03-21T08:44:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35799.log: /var/log/ceph/ceph-client.admin.32286.log 2024-03-21T08:44:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34188.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-03-21T08:44:56.639 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34188.log.gz 2024-03-21T08:44:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-03-21T08:44:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-21T08:44:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33191.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32286.log.gz 2024-03-21T08:44:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36758.log 2024-03-21T08:44:56.642 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-03-21T08:44:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37167.log 2024-03-21T08:44:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41065.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-21T08:44:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36428.log 2024-03-21T08:44:56.644 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36758.log.gz 2024-03-21T08:44:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32313.log 2024-03-21T08:44:56.645 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37167.log.gz 2024-03-21T08:44:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38702.log 2024-03-21T08:44:56.646 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36428.log.gz 2024-03-21T08:44:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2024-03-21T08:44:56.647 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32313.log.gz 2024-03-21T08:44:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36532.log 2024-03-21T08:44:56.649 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38702.log.gz 2024-03-21T08:44:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32417.log 2024-03-21T08:44:56.649 INFO:teuthology.orchestra.run.smithi151.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-21T08:44:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37798.log: /var/log/ceph/ceph-client.admin.36532.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 2024-03-21T08:44:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36324.log 2024-03-21T08:44:56.651 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36532.log.gz 2024-03-21T08:44:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33774.log 2024-03-21T08:44:56.652 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32417.log.gz 2024-03-21T08:44:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32552.log 2024-03-21T08:44:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36324.log: /var/log/ceph/ceph-client.admin.33774.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36324.log.gz 2024-03-21T08:44:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31580.log 2024-03-21T08:44:56.654 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33774.log.gz 2024-03-21T08:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36454.log 2024-03-21T08:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32552.log.gz 2024-03-21T08:44:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-03-21T08:44:56.656 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31580.log.gz 2024-03-21T08:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31796.log 2024-03-21T08:44:56.657 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36454.log.gz 2024-03-21T08:44:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36968.log 2024-03-21T08:44:56.658 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-03-21T08:44:56.659 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34896.log 2024-03-21T08:44:56.659 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31796.log.gz 2024-03-21T08:44:56.660 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33540.log 2024-03-21T08:44:56.660 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36968.log.gz 2024-03-21T08:44:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-03-21T08:44:56.661 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34896.log.gz 2024-03-21T08:44:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37981.log 2024-03-21T08:44:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33540.log: /var/log/ceph/ceph-client.admin.37404.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33540.log.gz 2024-03-21T08:44:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40825.log 2024-03-21T08:44:56.663 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-03-21T08:44:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37824.log 2024-03-21T08:44:56.664 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37981.log.gz 2024-03-21T08:44:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-03-21T08:44:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40825.log: /var/log/ceph/ceph-client.admin.37824.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40825.log.gz 2024-03-21T08:44:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32102.log 2024-03-21T08:44:56.666 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37824.log.gz 2024-03-21T08:44:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39366.log/var/log/ceph/ceph-client.admin.33384.log: 2024-03-21T08:44:56.667 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz/var/log/ceph/ceph-client.admin.32102.log: 2024-03-21T08:44:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2024-03-21T08:44:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-03-21T08:44:56.670 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39366.log.gz 2024-03-21T08:44:56.670 INFO:teuthology.orchestra.run.smithi028.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.32102.log.gz 2024-03-21T08:44:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35773.log 2024-03-21T08:44:56.671 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz 2024-03-21T08:44:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37720.log 2024-03-21T08:44:56.672 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-03-21T08:44:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-03-21T08:44:56.673 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35773.log.gz 2024-03-21T08:44:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-03-21T08:44:56.674 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37720.log.gz 2024-03-21T08:44:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32770.log 2024-03-21T08:44:56.675 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35352.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-03-21T08:44:56.675 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-03-21T08:44:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31502.log 2024-03-21T08:44:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log/var/log/ceph/ceph-client.admin.32770.log: 2024-03-21T08:44:56.677 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4%/var/log/ceph/ceph-client.admin.31502.log: -- replaced with /var/log/ceph/ceph-client.admin.32770.log.gz 2024-03-21T08:44:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-03-21T08:44:56.678 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31502.log.gz 2024-03-21T08:44:56.678 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-03-21T08:44:56.679 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-03-21T08:44:56.679 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36064.log 2024-03-21T08:44:56.680 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-03-21T08:44:56.680 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34662.log 2024-03-21T08:44:56.681 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-03-21T08:44:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41120.log 2024-03-21T08:44:56.682 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36064.log.gz 2024-03-21T08:44:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37085.log 2024-03-21T08:44:56.683 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34662.log.gz 2024-03-21T08:44:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.41120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31744.log 2024-03-21T08:44:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41826.log 2024-03-21T08:44:56.685 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.41120.log.gz 2024-03-21T08:44:56.685 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37085.log.gz 2024-03-21T08:44:56.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35534.log 2024-03-21T08:44:56.686 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31744.log.gz 2024-03-21T08:44:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33592.log 2024-03-21T08:44:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41826.log: /var/log/ceph/ceph-client.admin.35534.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41826.log.gz 2024-03-21T08:44:56.688 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35274.log 2024-03-21T08:44:56.688 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35534.log.gz 2024-03-21T08:44:56.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37480.log 2024-03-21T08:44:56.689 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33592.log.gz 2024-03-21T08:44:56.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34428.log 2024-03-21T08:44:56.690 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35274.log.gz 2024-03-21T08:44:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-03-21T08:44:56.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37480.log.gz 2024-03-21T08:44:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35851.log 2024-03-21T08:44:56.692 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34428.log.gz 2024-03-21T08:44:56.693 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2024-03-21T08:44:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39061.log: /var/log/ceph/ceph-client.admin.35851.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-03-21T08:44:56.694 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35851.log.gz 2024-03-21T08:44:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38465.log 2024-03-21T08:44:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35666.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-21T08:44:56.696 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2024-03-21T08:44:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38007.log 2024-03-21T08:44:56.697 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38465.log.gz 2024-03-21T08:44:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39900.log 2024-03-21T08:44:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-03-21T08:44:56.699 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38007.log.gz 2024-03-21T08:44:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31690.log 2024-03-21T08:44:56.700 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39900.log.gz 2024-03-21T08:44:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33696.log 2024-03-21T08:44:56.702 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-03-21T08:44:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-03-21T08:44:56.702 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31690.log.gz 2024-03-21T08:44:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38809.log 2024-03-21T08:44:56.703 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33696.log.gz 2024-03-21T08:44:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31554.log 2024-03-21T08:44:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35640.log: 87.8%/var/log/ceph/ceph-client.admin.38809.log: -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-03-21T08:44:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32578.log 2024-03-21T08:44:56.705 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38809.log.gz 2024-03-21T08:44:56.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31554.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31554.log.gz 2024-03-21T08:44:56.706 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-03-21T08:44:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.32578.log: --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-03-21T08:44:56.708 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32578.log.gz 2024-03-21T08:44:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-03-21T08:44:56.709 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-03-21T08:44:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32443.log 2024-03-21T08:44:56.710 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-03-21T08:44:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36642.log 2024-03-21T08:44:56.711 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-03-21T08:44:56.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32182.log 2024-03-21T08:44:56.712 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32443.log.gz 2024-03-21T08:44:56.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2024-03-21T08:44:56.713 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36642.log.gz 2024-03-21T08:44:56.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33828.log 2024-03-21T08:44:56.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32182.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32182.log.gz 2024-03-21T08:44:56.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33462.log 2024-03-21T08:44:56.718 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32156.log 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.33828.log.gz 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39785.log 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33462.log.gz 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32208.log 2024-03-21T08:44:56.719 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32156.log.gz 2024-03-21T08:44:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 87.9% 2024-03-21T08:44:56.720 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39785.log.gz 2024-03-21T08:44:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34506.log 2024-03-21T08:44:56.721 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32208.log.gz 2024-03-21T08:44:56.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33748.log 2024-03-21T08:44:56.722 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2024-03-21T08:44:56.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2024-03-21T08:44:56.723 INFO:teuthology.orchestra.run.smithi028.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34506.log.gz 2024-03-21T08:44:56.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33748.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-21T08:44:56.724 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33748.log.gz 2024-03-21T08:44:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-03-21T08:44:56.725 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2024-03-21T08:44:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2024-03-21T08:44:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2024-03-21T08:44:56.728 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-03-21T08:44:56.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37637.log 2024-03-21T08:44:56.729 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2024-03-21T08:44:56.729 INFO:teuthology.orchestra.run.smithi151.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-21T08:44:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32981.log 2024-03-21T08:44:56.730 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2024-03-21T08:44:56.731 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37637.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37637.log.gz 2024-03-21T08:44:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32981.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32981.log.gz 2024-03-21T08:44:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32777.log 2024-03-21T08:44:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37929.log 2024-03-21T08:44:56.735 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37297.log 2024-03-21T08:44:56.735 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32777.log.gz 2024-03-21T08:44:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37929.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37929.log.gz 2024-03-21T08:44:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37297.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37297.log.gz 2024-03-21T08:44:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-03-21T08:44:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35008.log 2024-03-21T08:44:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33436.log 2024-03-21T08:44:56.741 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-03-21T08:44:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35008.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35008.log.gz 2024-03-21T08:44:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33436.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33436.log.gz 2024-03-21T08:44:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36116.log 2024-03-21T08:44:56.745 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32974.log 2024-03-21T08:44:56.745 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-03-21T08:44:56.746 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36116.log.gz/var/log/ceph/ceph-client.admin.32974.log: 2024-03-21T08:44:56.747 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32974.log.gz 2024-03-21T08:44:56.748 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32708.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-03-21T08:44:56.748 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35113.log 2024-03-21T08:44:56.750 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-03-21T08:44:56.751 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39587.log 2024-03-21T08:44:56.751 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35113.log.gz 2024-03-21T08:44:56.752 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40956.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-03-21T08:44:56.753 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39587.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39587.log.gz 2024-03-21T08:44:56.753 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-03-21T08:44:56.755 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34714.log 2024-03-21T08:44:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41607.log 2024-03-21T08:44:56.757 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-03-21T08:44:56.757 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34714.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34714.log.gz 2024-03-21T08:44:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41529.log 2024-03-21T08:44:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41607.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41607.log.gz 2024-03-21T08:44:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35482.log 2024-03-21T08:44:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41529.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41529.log.gz 2024-03-21T08:44:56.761 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40799.log 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35482.log.gz 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40799.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40799.log.gz 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41034.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34053.log 2024-03-21T08:44:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39848.log 2024-03-21T08:44:56.768 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-21T08:44:56.769 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34053.log.gz 2024-03-21T08:44:56.769 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39848.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39848.log.gz 2024-03-21T08:44:56.769 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40088.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-21T08:44:56.770 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41477.log 2024-03-21T08:44:56.771 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36272.log 2024-03-21T08:44:56.772 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-03-21T08:44:56.773 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41477.log.gz 2024-03-21T08:44:56.774 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34844.log 2024-03-21T08:44:56.774 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36272.log.gz 2024-03-21T08:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37430.log: /var/log/ceph/ceph-client.admin.34844.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-03-21T08:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34844.log.gz 2024-03-21T08:44:56.777 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.42086.log 2024-03-21T08:44:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38225.log 2024-03-21T08:44:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40609.log 2024-03-21T08:44:56.779 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-03-21T08:44:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38225.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38225.log.gz 2024-03-21T08:44:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40609.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40609.log.gz 2024-03-21T08:44:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37352.log 2024-03-21T08:44:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32787.log 2024-03-21T08:44:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33192.log 2024-03-21T08:44:56.785 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37352.log.gz 2024-03-21T08:44:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32787.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32787.log.gz 2024-03-21T08:44:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33192.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33192.log.gz 2024-03-21T08:44:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-03-21T08:44:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27301.log 2024-03-21T08:44:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35456.log 2024-03-21T08:44:56.790 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-03-21T08:44:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27301.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27301.log.gz 2024-03-21T08:44:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35456.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35456.log.gz 2024-03-21T08:44:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38147.log 2024-03-21T08:44:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2024-03-21T08:44:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36610.log 2024-03-21T08:44:56.795 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38147.log.gz 2024-03-21T08:44:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41233.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2024-03-21T08:44:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36610.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36610.log.gz 2024-03-21T08:44:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33644.log 2024-03-21T08:44:56.798 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-21T08:44:56.799 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31716.log 2024-03-21T08:44:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34818.log 2024-03-21T08:44:56.800 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33644.log.gz 2024-03-21T08:44:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31716.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31716.log.gz 2024-03-21T08:44:56.802 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34818.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34818.log.gz 2024-03-21T08:44:56.802 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-03-21T08:44:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42398.log 2024-03-21T08:44:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33566.log 2024-03-21T08:44:56.805 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-03-21T08:44:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42398.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42398.log.gz 2024-03-21T08:44:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33566.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33566.log.gz 2024-03-21T08:44:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37663.log 2024-03-21T08:44:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-21T08:44:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36838.log 2024-03-21T08:44:56.810 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37663.log.gz 2024-03-21T08:44:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.36838.log: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-21T08:44:56.813 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36838.log.gz 2024-03-21T08:44:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-21T08:44:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33176.log 2024-03-21T08:44:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-03-21T08:44:56.816 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-21T08:44:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33176.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33176.log.gz 2024-03-21T08:44:56.817 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38356.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz 2024-03-21T08:44:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34272.log 2024-03-21T08:44:56.819 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31351.log 2024-03-21T08:44:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-03-21T08:44:56.821 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34272.log.gz 2024-03-21T08:44:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31351.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31351.log.gz 2024-03-21T08:44:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35326.log 2024-03-21T08:44:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36012.log 2024-03-21T08:44:56.823 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-03-21T08:44:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32773.log 2024-03-21T08:44:56.825 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35326.log.gz 2024-03-21T08:44:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36012.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36012.log.gz 2024-03-21T08:44:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32773.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32773.log.gz 2024-03-21T08:44:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2024-03-21T08:44:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-03-21T08:44:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36480.log 2024-03-21T08:44:56.830 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2024-03-21T08:44:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35378.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-03-21T08:44:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36480.log.gz 2024-03-21T08:44:56.832 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38835.log 2024-03-21T08:44:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34214.log 2024-03-21T08:44:56.835 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-03-21T08:44:56.835 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38835.log.gz 2024-03-21T08:44:56.835 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34214.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34214.log.gz 2024-03-21T08:44:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41347.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-03-21T08:44:56.837 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31449.log 2024-03-21T08:44:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36402.log 2024-03-21T08:44:56.840 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-03-21T08:44:56.840 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31449.log.gz 2024-03-21T08:44:56.840 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36402.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36402.log.gz 2024-03-21T08:44:56.841 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32604.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-03-21T08:44:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2024-03-21T08:44:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37611.log 2024-03-21T08:44:56.844 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-03-21T08:44:56.845 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2024-03-21T08:44:56.846 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37611.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37611.log.gz 2024-03-21T08:44:56.846 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39191.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-03-21T08:44:56.847 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-21T08:44:56.848 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36916.log 2024-03-21T08:44:56.850 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39470.log 2024-03-21T08:44:56.850 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-21T08:44:56.851 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36916.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36916.log.gz 2024-03-21T08:44:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39470.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39470.log.gz 2024-03-21T08:44:56.852 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2024-03-21T08:44:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-03-21T08:44:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-03-21T08:44:56.855 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz 2024-03-21T08:44:56.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41930.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-03-21T08:44:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40667.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-03-21T08:44:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-03-21T08:44:56.859 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32980.log 2024-03-21T08:44:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36864.log 2024-03-21T08:44:56.861 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-03-21T08:44:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32980.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32980.log.gz 2024-03-21T08:44:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36864.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36864.log.gz 2024-03-21T08:44:56.863 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32996.log 2024-03-21T08:44:56.864 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37535.log 2024-03-21T08:44:56.865 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-03-21T08:44:56.866 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32996.log.gz 2024-03-21T08:44:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.37535.log: /var/log/ceph/ceph-client.admin.32500.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2024-03-21T08:44:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2024-03-21T08:44:56.868 INFO:teuthology.orchestra.run.smithi028.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.37535.log.gz 2024-03-21T08:44:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40114.log 2024-03-21T08:44:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42008.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-03-21T08:44:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40114.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40114.log.gz 2024-03-21T08:44:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34298.log 2024-03-21T08:44:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35877.log 2024-03-21T08:44:56.875 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-03-21T08:44:56.875 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34298.log.gz 2024-03-21T08:44:56.876 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35877.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35877.log.gz 2024-03-21T08:44:56.877 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38251.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-03-21T08:44:56.877 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27416.log 2024-03-21T08:44:56.878 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34792.log 2024-03-21T08:44:56.879 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.27416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2024-03-21T08:44:56.880 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27416.log.gz 2024-03-21T08:44:56.881 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34792.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34792.log.gz 2024-03-21T08:44:56.882 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34688.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2024-03-21T08:44:56.882 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35191.log 2024-03-21T08:44:56.883 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40531.log 2024-03-21T08:44:56.885 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-03-21T08:44:56.885 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35191.log.gz 2024-03-21T08:44:56.886 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40531.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40531.log.gz 2024-03-21T08:44:56.887 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41311.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2024-03-21T08:44:56.887 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32781.log 2024-03-21T08:44:56.889 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36038.log 2024-03-21T08:44:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37746.log 2024-03-21T08:44:56.890 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32781.log.gz 2024-03-21T08:44:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36038.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36038.log.gz 2024-03-21T08:44:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37746.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37746.log.gz 2024-03-21T08:44:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-03-21T08:44:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32234.log 2024-03-21T08:44:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41704.log 2024-03-21T08:44:56.896 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-03-21T08:44:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32234.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32234.log.gz 2024-03-21T08:44:56.897 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41704.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41704.log.gz 2024-03-21T08:44:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32260.log 2024-03-21T08:44:56.899 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32984.log 2024-03-21T08:44:56.901 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37141.log 2024-03-21T08:44:56.901 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32260.log.gz 2024-03-21T08:44:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32984.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32984.log.gz 2024-03-21T08:44:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37141.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37141.log.gz 2024-03-21T08:44:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41798.log 2024-03-21T08:44:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38983.log 2024-03-21T08:44:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-03-21T08:44:56.906 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41798.log.gz 2024-03-21T08:44:56.907 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38983.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38983.log.gz 2024-03-21T08:44:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39113.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-03-21T08:44:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37219.log 2024-03-21T08:44:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-03-21T08:44:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32975.log 2024-03-21T08:44:56.911 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37219.log.gz 2024-03-21T08:44:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39522.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-03-21T08:44:56.913 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32975.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32975.log.gz 2024-03-21T08:44:56.914 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2024-03-21T08:44:56.915 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2024-03-21T08:44:56.916 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36786.log 2024-03-21T08:44:56.917 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2024-03-21T08:44:56.917 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40851.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2024-03-21T08:44:56.918 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36786.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36786.log.gz 2024-03-21T08:44:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-21T08:44:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33889.log 2024-03-21T08:44:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34454.log 2024-03-21T08:44:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33889.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33889.log.gz 2024-03-21T08:44:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34454.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-21T08:44:56.924 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34454.log.gz 2024-03-21T08:44:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-21T08:44:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36890.log 2024-03-21T08:44:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37193.log 2024-03-21T08:44:56.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36890.log.gz 2024-03-21T08:44:56.928 INFO:teuthology.orchestra.run.smithi028.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-21T08:44:56.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37193.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37193.log.gz 2024-03-21T08:44:56.929 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-21T08:44:56.930 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37903.log 2024-03-21T08:44:56.932 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37877.log 2024-03-21T08:44:56.933 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37903.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37903.log.gz 2024-03-21T08:44:56.934 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2024-03-21T08:44:56.935 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37877.log.gz 2024-03-21T08:44:56.935 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38382.log 2024-03-21T08:44:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42164.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2024-03-21T08:44:56.937 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38382.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38382.log.gz 2024-03-21T08:44:56.938 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-21T08:44:56.939 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34766.log 2024-03-21T08:44:56.941 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40166.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-03-21T08:44:56.941 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34766.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34766.log.gz 2024-03-21T08:44:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35404.log 2024-03-21T08:44:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2024-03-21T08:44:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35404.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35404.log.gz 2024-03-21T08:44:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38330.log 2024-03-21T08:44:56.946 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2024-03-21T08:44:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38039.log 2024-03-21T08:44:56.949 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38330.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38330.log.gz 2024-03-21T08:44:56.949 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38039.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38039.log.gz 2024-03-21T08:44:56.950 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-03-21T08:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34009.log 2024-03-21T08:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39217.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-03-21T08:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.34009.log: 90.3% -- replaced with /var/log/ceph/ceph-client.0.34009.log.gz 2024-03-21T08:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39243.log 2024-03-21T08:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2024-03-21T08:44:56.957 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39243.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39243.log.gz 2024-03-21T08:44:56.958 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41008.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2024-03-21T08:44:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-03-21T08:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32656.log 2024-03-21T08:44:56.962 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36142.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-03-21T08:44:56.962 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32656.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32656.log.gz 2024-03-21T08:44:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36090.log 2024-03-21T08:44:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-03-21T08:44:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36090.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36090.log.gz 2024-03-21T08:44:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39874.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-03-21T08:44:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35960.log 2024-03-21T08:44:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39314.log 2024-03-21T08:44:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35960.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35960.log.gz 2024-03-21T08:44:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40323.log 2024-03-21T08:44:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39314.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39314.log.gz 2024-03-21T08:44:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34584.log 2024-03-21T08:44:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.40323.log: /var/log/ceph/ceph-client.admin.34584.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.40323.log.gz 2024-03-21T08:44:56.975 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34584.log.gz 2024-03-21T08:44:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42112.log 2024-03-21T08:44:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37059.log 2024-03-21T08:44:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42112.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42112.log.gz 2024-03-21T08:44:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37059.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37059.log.gz 2024-03-21T08:44:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2024-03-21T08:44:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35217.log 2024-03-21T08:44:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40036.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2024-03-21T08:44:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37772.log 2024-03-21T08:44:56.983 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35217.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35217.log.gz 2024-03-21T08:44:56.984 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-03-21T08:44:56.985 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37772.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37772.log.gz 2024-03-21T08:44:56.986 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39703.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-03-21T08:44:56.987 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37850.log 2024-03-21T08:44:56.988 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32784.log 2024-03-21T08:44:56.989 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37850.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37850.log.gz 2024-03-21T08:44:56.990 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32784.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32784.log.gz 2024-03-21T08:44:56.991 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32339.log 2024-03-21T08:44:56.991 INFO:teuthology.orchestra.run.smithi028.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-21T08:44:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32785.log 2024-03-21T08:44:56.992 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-21T08:44:56.993 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38781.log 2024-03-21T08:44:56.993 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32339.log.gz 2024-03-21T08:44:56.994 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38491.log 2024-03-21T08:44:56.994 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32785.log.gz 2024-03-21T08:44:56.995 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-03-21T08:44:56.995 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38781.log.gz 2024-03-21T08:44:56.996 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2024-03-21T08:44:56.997 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38491.log.gz 2024-03-21T08:44:56.997 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41756.log 2024-03-21T08:44:56.997 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-03-21T08:44:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38546.log 2024-03-21T08:44:56.998 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-03-21T08:44:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40426.log 2024-03-21T08:44:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41756.log: /var/log/ceph/ceph-client.0.38546.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2024-03-21T08:44:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37955.log 2024-03-21T08:44:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34324.log 2024-03-21T08:44:57.003 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40426.log.gz 2024-03-21T08:44:57.003 INFO:teuthology.orchestra.run.smithi028.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.38546.log.gz 2024-03-21T08:44:57.003 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2024-03-21T08:44:57.003 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37955.log.gz 2024-03-21T08:44:57.004 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-03-21T08:44:57.004 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34324.log: 88.2%/var/log/ceph/ceph-client.admin.41373.log: -- replaced with /var/log/ceph/ceph-client.admin.34324.log.gz 2024-03-21T08:44:57.005 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38598.log 2024-03-21T08:44:57.005 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-03-21T08:44:57.005 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36706.log 2024-03-21T08:44:57.006 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-03-21T08:44:57.007 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33211.log 2024-03-21T08:44:57.007 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38598.log.gz 2024-03-21T08:44:57.008 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38755.log 2024-03-21T08:44:57.008 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36706.log.gz 2024-03-21T08:44:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-03-21T08:44:57.009 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33211.log.gz 2024-03-21T08:44:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-03-21T08:44:57.010 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38755.log.gz 2024-03-21T08:44:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37378.log 2024-03-21T08:44:57.011 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-03-21T08:44:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-03-21T08:44:57.012 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-03-21T08:44:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33185.log 2024-03-21T08:44:57.013 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37378.log.gz 2024-03-21T08:44:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-03-21T08:44:57.015 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-03-21T08:44:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31976.log 2024-03-21T08:44:57.016 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33185.log.gz 2024-03-21T08:44:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35614.log 2024-03-21T08:44:57.017 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-03-21T08:44:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37323.log 2024-03-21T08:44:57.018 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31976.log.gz 2024-03-21T08:44:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35508.log 2024-03-21T08:44:57.019 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35614.log.gz 2024-03-21T08:44:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38676.log 2024-03-21T08:44:57.020 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37323.log.gz 2024-03-21T08:44:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37271.log 2024-03-21T08:44:57.021 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35508.log.gz 2024-03-21T08:44:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.38676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-03-21T08:44:57.022 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38676.log.gz 2024-03-21T08:44:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31770.log 2024-03-21T08:44:57.024 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37271.log.gz 2024-03-21T08:44:57.024 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-03-21T08:44:57.024 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-03-21T08:44:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40504.log 2024-03-21T08:44:57.026 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31770.log.gz 2024-03-21T08:44:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-03-21T08:44:57.027 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-03-21T08:44:57.027 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31384.log 2024-03-21T08:44:57.028 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40504.log.gz 2024-03-21T08:44:57.029 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2024-03-21T08:44:57.029 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-21T08:44:57.029 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41425.log 2024-03-21T08:44:57.030 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31384.log.gz 2024-03-21T08:44:57.031 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2024-03-21T08:44:57.031 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2024-03-21T08:44:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-03-21T08:44:57.032 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41425.log.gz 2024-03-21T08:44:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-03-21T08:44:57.033 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2024-03-21T08:44:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33618.log 2024-03-21T08:44:57.034 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-03-21T08:44:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36168.log 2024-03-21T08:44:57.035 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-03-21T08:44:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-03-21T08:44:57.037 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33618.log.gz 2024-03-21T08:44:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2024-03-21T08:44:57.037 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36168.log.gz 2024-03-21T08:44:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-03-21T08:44:57.039 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-03-21T08:44:57.040 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35934.log 2024-03-21T08:44:57.040 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gz 2024-03-21T08:44:57.041 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36558.log 2024-03-21T08:44:57.041 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-03-21T08:44:57.042 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38094.log 2024-03-21T08:44:57.042 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35934.log.gz 2024-03-21T08:44:57.043 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33410.log 2024-03-21T08:44:57.043 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36558.log.gz 2024-03-21T08:44:57.044 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.38094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32991.log 2024-03-21T08:44:57.045 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33410.log: 91.0% -- replaced with /var/log/ceph/ceph-client.0.38094.log.gz 2024-03-21T08:44:57.045 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35300.log 2024-03-21T08:44:57.046 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33410.log.gz 2024-03-21T08:44:57.046 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.32991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31475.log 2024-03-21T08:44:57.047 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32991.log.gz 2024-03-21T08:44:57.048 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33670.log 2024-03-21T08:44:57.048 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35300.log.gz 2024-03-21T08:44:57.048 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.31475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35908.log 2024-03-21T08:44:57.049 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31475.log.gz 2024-03-21T08:44:57.049 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.33670.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33670.log.gz 2024-03-21T08:44:57.051 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.35908.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35908.log.gz 2024-03-21T08:44:57.285 INFO:teuthology.orchestra.run.smithi151.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-21T08:44:57.319 INFO:teuthology.orchestra.run.smithi151.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-21T08:44:57.332 INFO:teuthology.orchestra.run.smithi151.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-21T08:44:57.414 INFO:teuthology.orchestra.run.smithi151.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-21T08:44:57.416 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-21T08:44:57.417 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m0.878s 2024-03-21T08:44:57.417 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m3.273s 2024-03-21T08:44:57.417 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.310s 2024-03-21T08:44:57.454 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-21T08:44:57.457 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-21T08:44:57.555 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-21T08:44:57.620 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-21T08:44:57.622 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-21T08:44:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m1.107s 2024-03-21T08:44:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m4.128s 2024-03-21T08:44:57.623 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.901s 2024-03-21T08:44:57.624 INFO:tasks.ceph:Archiving logs... 2024-03-21T08:44:57.625 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi028/log 2024-03-21T08:44:57.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T08:44:58.399 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi151/log 2024-03-21T08:44:58.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T08:44:58.829 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-21T08:44:58.840 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-21T08:44:58.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T08:44:58.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T08:44:58.926 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T08:44:58.936 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T08:44:58.936 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T08:44:58.940 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T08:44:58.952 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2024-03-21T08:44:58.952 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2024-03-21T08:44:58.952 INFO:teuthology.orchestra.run.smithi028.stdout:*hv01.front.sepi 44.4.53.2 2 u 57 64 177 0.098 -0.722 1.412 2024-03-21T08:44:58.953 INFO:teuthology.orchestra.run.smithi028.stdout:+hv02.front.sepi 96.245.170.99 3 u 61 64 177 0.088 -1.382 1.519 2024-03-21T08:44:58.953 INFO:teuthology.orchestra.run.smithi028.stdout:+hv03.front.sepi 74.6.168.72 3 u 59 64 177 0.129 -2.209 1.554 2024-03-21T08:44:58.953 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:44:58.956 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2024-03-21T08:44:58.956 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2024-03-21T08:44:58.956 INFO:teuthology.orchestra.run.smithi151.stdout:*hv01.front.sepi 44.4.53.2 2 u 56 64 177 0.074 -0.011 0.435 2024-03-21T08:44:58.957 INFO:teuthology.orchestra.run.smithi151.stdout:+hv02.front.sepi 96.245.170.99 3 u 61 64 177 0.057 -1.026 0.433 2024-03-21T08:44:58.957 INFO:teuthology.orchestra.run.smithi151.stdout:+hv03.front.sepi 74.6.168.72 3 u 58 64 177 0.077 -1.841 0.446 2024-03-21T08:44:58.957 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-21T08:44:58.958 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T08:44:58.972 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T08:44:58.972 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T08:44:58.992 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T08:44:59.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T08:44:59.059 INFO:teuthology.task.internal:Duration was 938.398613 seconds 2024-03-21T08:44:59.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T08:44:59.093 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T08:44:59.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T08:44:59.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T08:44:59.190 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T08:44:59.190 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-03-21T08:44:59.191 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T08:44:59.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-21T08:44:59.212 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T08:44:59.232 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T08:44:59.232 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T08:44:59.260 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T08:44:59.313 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T08:44:59.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T08:44:59.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T08:44:59.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T08:44:59.444 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T08:44:59.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T08:44:59.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T08:44:59.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T08:44:59.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T08:44:59.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T08:44:59.527 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-03-21T08:44:59.545 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-21T08:44:59.564 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T08:44:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:59.596 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T08:44:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T08:44:59.611 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T08:44:59.621 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T08:44:59.622 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi028 2024-03-21T08:44:59.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T08:44:59.665 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126/remote/smithi151 2024-03-21T08:44:59.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T08:44:59.690 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T08:44:59.690 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T08:44:59.704 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T08:44:59.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T08:44:59.752 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T08:44:59.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T08:44:59.777 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T08:44:59.777 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T08:44:59.780 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T08:44:59.786 INFO:teuthology.orchestra.run.smithi028.stdout: 658117 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 08:44 /home/ubuntu/cephtest 2024-03-21T08:44:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: 658116 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 08:44 /home/ubuntu/cephtest 2024-03-21T08:44:59.798 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T08:44:59.811 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T08:44:59.877 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T08:44:59.905 DEBUG:teuthology.nuke:shortname: smithi028 2024-03-21T08:44:59.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T08:44:59.929 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-21T08:44:59.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T08:44:59.933 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 08:16:03.359200', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqM1DoNJIt2li0uTk/Ne4B6R5ZL8dWRK89+IfGdoWYkOgN7cBCrLUdVrRidmATZeTEUnWuEUd9RvhI3Jf/GsMc='} 2024-03-21T08:44:59.957 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-20_11:21:18-fs-wip-vshankar-testing-20240320.053002-testing-default-smithi/7612126', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 08:16:03.362281', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVQ4EoPkwqcFmwRr8rgNQDfaP1UqmopUAgnGlaKXBO7f187Ba+V0DzAY2sUYrJcKh7VjUHBywJPbUrQgUqXU2w='} 2024-03-21T08:44:59.985 INFO:teuthology.orchestra.console.smithi028:Power off 2024-03-21T08:44:59.986 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T08:45:00.011 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-03-21T08:45:00.011 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:00.030 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-21T08:45:00.031 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T08:45:00.041 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-21T08:45:00.063 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-21T08:45:00.063 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:00.092 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-21T08:45:04.043 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:04.093 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:04.154 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-21T08:45:04.205 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-21T08:45:08.159 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:08.207 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:08.270 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-03-21T08:45:08.320 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-21T08:45:08.371 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-03-21T08:45:12.322 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T08:45:12.435 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-21T08:45:12.536 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-21T08:45:12.641 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} duration: 938.3986127376556 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default mon.a-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 mon.b-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=75db766259d14ced947da6222804275b status: fail success: false 2024-03-21T08:45:12.642 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T08:45:12.731 INFO:teuthology.run:FAIL